2006, 2019

Kafka 05: Kafka Consumer with ElasticSearch

By |June 20th, 2019|Categories: Technology|

Pre Configure 1. Register Bonsai: 2. Increase maximum limit: (We will demostrate batch process, so 1000 is really easy to fulfil) [crayon-5d3510103c46e269891047/] 3. Install Maven dependency [crayon-5d3510103c476969662705/]     The Real Code 1. Setup ElasticSearch Client [crayon-5d3510103c479589357656/] 2. Create a Kafka Consumer [crayon-5d3510103c47c984703259/] 3. Post records from Kafka into ElasticSearch (Bonsai) [crayon-5d3510103c47e153236326/]   Consumer Offset Commit Strategy Things about at [...]

1706, 2019

PHP xDebug with Docker

By |June 17th, 2019|Categories: Technology|

Single PC with Remote Debug   Step 1: Install xDebug in remote web server.  Step 2: Modify xdebug.ini (or php.ini) [crayon-5d3510103d6f6703391284/] Step 3: Setup PHPStorm setup File -> Settings -> Languages & Frameworks -> PHP -> CLI Interpreter -> ... -> + from Docker (one project should be Frontend, one project is Backend, they are separate.) setup File -> Settings -> Languages & Frameworks -> PHP -> Debug. [...]

1206, 2019

Kafka 04: Producer Configurations

By |June 12th, 2019|Categories: Technology|Tags: |

Overview The overview of our TwitterProducer config. [crayon-5d3510103d9ba450306685/]     About acks [crayon-5d3510103d9c4666612115/] acks = 0 no response is required if the broker goes offline, we will lose data useful for data where it's ok to lose: metrics, log collection. acks = 1 leader response is requested. no replication is required. the producer may retry, if ack from leader is not received if [...]

1106, 2019

Kafka 03: Twitter Producer (Java)

By |June 11th, 2019|Categories: Technology|Tags: |

Setup Twitter Developer Account and Create an App link: You need to give good reason and detail your app description.     Get Dependencies link: [crayon-5d3510103dcf3561116657/] Create New Producer and Consumer Step 1: Overview [crayon-5d3510103dcfa172528116/] Step 2: Create a Twitter Client [crayon-5d3510103dcfd837337607/] Step 3: Create a Kafka Producer [crayon-5d3510103dcff631548996/] Step 4: Create a Topic [crayon-5d3510103dd02589522242/] [...]

2705, 2019

Kafka 02: Install and CLI

By |May 27th, 2019|Categories: Technology|Tags: |

Install Download and unzip unzip and unzip paste under C:\ Create a folder data under kafka's root dir. Create two folders: kafka and zookeeper under data folder. Change properties: config\ dataDir=C:/kafka_2.12-2.2.0/data/zookeeper config\ log.dirs=C:/kafka_2.12-2.2.0/data/kafka   Launch Kafka Add environment PATH: C:\kafka_2.12-2.2.0\bin\windows Launch Zookeeper FIRST, then start Kafka: zookeeper-server-start.bat  config\ kafka-server-start.bat  config\ kafka-topics 1. Create C:\kafka_2.12-2.2.0>kafka-topics --zookeeper [...]

2205, 2019

Kafka 01: The Theory

By |May 22nd, 2019|Categories: Technology|Tags: |

Topics, Partitions and Offsets Topics: A particular stream of data. Similar to a table in a database. You can have as many topics as you want. It is identified by name Partitions: It splits a topic. Each partition is ordered. When you create a topic, you have to define how many partitions you want. Offset: Within a partition, each message gets an auto incremental id. It's infinite. Offset only [...]