Preparation

Preparation

Our example use case simulates sales data coming from the field, and being processed via a Kinesis stream and written out into ElasticSearch where we can analyse the sales data graphically with Kibana. The sales data will be read from a CSV file, and have costs, quantities and transaction dates randomly applied. The architecture looks like this: Prequistite

Create a new repository for Kinesis Consumer source

  1. Go to AWS CodeCommit console.
  • Click Repositories
  • Click Create repository Prequistite
  1. In the Create repository page
  • In the Repository name section, type KinesisConsumer
  • In the Description section, type Sample application that consumes messages from a Kinesis stream and writes them to an ElasticSearch domain index
  • Click Create Prequistite
  1. In the KinesisConsumer page, click Clone URL
  • Click Clone HTTPS to save the url Prequistite
  1. In the Eclipse IDE, click Window
  • Click Perspective
  • Click Open Perspective
  • Click Other Prequistite
  1. In the Open Perspective section, click Git
  • Click Open Prequistite
  1. Click Clone a Git repository Prequistite
  2. In the Source Git Repository section
  • In the URI section, type the url we saved in step 3 Prequistite
  1. Go to Amazon CloudFormation Console.
  • Click Stacks
  • Type DevAx-05 to the search bar, press Enter
  • Click DevAx-05 Prequistite
  1. Click tab Output
  • Save the value of GitUserName and GitPassword Prequistite
  1. In the Eclipse IDE, type the value of User and Password we saved in step 9
  • Click Next Prequistite
  1. Click Next Prequistite
  2. Click Finish Prequistite
  • KinesisConsumer.zip (9 ko)
    1. Download file KinesisConsumer.zip and extract
    • Copy the contents of the exploded zip file across to the filesystem location where you have cloned the KinesisConsumer repository.
    1. In the Eclipse IDE, Click File
    • Click Open projects from filesystem… Prequistite
    1. Choose the path of the KinesisConsumer project
    • Click Finish Prequistite
    1. Open file KinesisConsumer\src\main\java\idevelop\demo\config.java Prequistite
    2. Go to Amazon CloudFormation Console.
    • Click Stacks
    • Type DevAx-05 to the search bar, press Enter
    • Click DevAx-05 Prequistite
    1. Click tab Output
    • Save the value of ElasticsearchEndpoint Prequistite
    1. In the Eclipse IDE, replace REPLACE_WITH_YOUR_ELASTIC_SEARCH_ENDPOINT by ElasticsearchEndpoint we saved in step 18
    • In the REGION section, replace ap-southeast-2 by your Region code
    • Save Prequistite
    1. Open Command Prompt and navigate to the directory of KinesisConsumer project
    • Execute the below command to create master branch
    git checkout -b master

    Prequistite 21. Execute the below command to add the new file

    git add .

    Prequistite 22. Execute the below command to commit the new file

    git commit -m "Initial commit"

    Prequistite 23. In the Eclipse IDE, right-click on KinesisConsumer repository

    • Click Push Branch `master`… Prequistite
    1. Type the value of User and Password we saved in step 9 to User section and Password section
    • Click Login Prequistite
    1. Click Preview Prequistite
    2. Click Push Prequistite
    3. Go to AWS CodeCommit console.
    • Click Repositories
    • Type KinesisConsumer to the search bar
    • Click KinesisConsumer Prequistite
    1. You will see the files that you have added into the repository listed in the browser view. Prequistite

    Clone the repository on the Kinesis Consumer EC2 instance

    The labs setup has provisioned an EC2 instance for you to run as the Consumer in this scenario. You will now connect to the EC2 instance, clone the repository, build and run the Consumer code.

    In order to securely connect into your cloud infrastructure, you will use SSH, the secure shell. SSH requires a public/private key-pair to be set up and installed on the client and server that will communicate via the SSH protocol. The EC2 instances have been launched with a public key. You need to retrieve the private key so you can use that to establish a secure connection. Click Download PEM. Save this private key file on your local machine.

    1. Open Command Prompt and execute the below command
    chmod 400 <path to your key-pair file>

    Prequistite 30. Execute the below command

    ssh -i <path to your key-pair file> ec2-user@<EC2InstanceKinesisConsumer>

    Replace <EC2InstanceKinesisConsumer> by the value of EC2InstanceKinesisConsumer in the Output tab of DevAx-05 stack

    Prequistite 31. Execute the below command

    git clone https://git-codecommit.<your_region>.amazonaws.com/v1/repos/KinesisConsumer

    Replace <your_region> by your Region

    • Type User and Password we saved in step 9 to Username section and Password section Prequistite
    1. Execute the below command
    cd KinesisConsumer
/opt/apache-maven-3.3.9/bin/mvn package shade:shade

    Prequistite 33. It will take a few moments to complete the build. Prequistite 34. When the build is complete, start the Kinesis Consumer application by executing the below command

    java -jar target/KinesisConsumerApp-1.0.0.jar

    Prequistite The application will start and wait for events to be published into the Kinesis stream. You should see no errors, and after a few moments, the console output should show Sleeping… since there are no records in the stream to process: Prequistite

    Create a new repository for the Kinesis Producer source code

    You will now repeat the process above, this time, creating a repository for the Kinesis Producer application, and cloning it on to the Kinesis Producer EC2 instance. Once this is complete, you can begin testing data transfer between the two machines, via Amazon Kinesis.

    The process of creating the repository is the same as the previous steps, but we repeat it here for convenience.

    1. Go to AWS CodeCommit console.
    • Click Repositories
    • Click Create repository
    1. In the Create repository page
    • In the Repository name section, type KinesisProducer
    • In the Description section, type Sample application that publishes sales events into a Kinesis stream
    • Click Create
    1. In the KinesisProducer page, click Clone URL
    • Click Clone HTTPS to save the url
    1. In the Eclipse IDE, click Clone a Git Repository and add the clone to this view icon Prequistite
    2. In the Source Git Repository section
    • In the URI section, type url we saved in step 37
    1. Go to Amazon CloudFormation Console.
    • Click Stacks
    • Type DevAx-05 to the search bar, press Enter
    • Click DevAx-05
    1. Click tab Output
    • Save the value of GitUserName and GitPassword
    1. In the Eclipse IDE, type the value of User and Password we saved in step 9
    • Click Next
    1. Click Next
    2. Click Finish
  • KinesisProducer.zip (6 ko)
    1. Download file KinesisProducer.zip and extract
    • Copy the contents of the exploded zip file across to the filesystem location where you have cloned the KinesisProducer repository.
    1. In the Eclipse IDE, Click File
    • Click Open projects from filesystem…
    1. Select the path of KinesisProducer project
    • Click Finish
    1. Open file KinesisProducer\src\main\java\idevelop\demo\KinesisProducerApp.java Prequistite
    2. In the line 69, replace AP_SOUTHEAST_2 by your Region
    • Save Prequistite
    1. Open Command Prompt and navigate to the directory of KinesisProducer
    • Execute the below command to create master branch
    git checkout -b master
    1. Execute the below command to add the new files
    git add .
    1. Execute the below command to commit the new files
    git commit -m "Initial commit"
    1. In the Eclipse IDE, right-click on KinesisProducer repository
    • Click Push Branch `master`…
    1. Type the value of User and Password we saved in step 41 to User section and Password section
    • Click Login
    1. Click Preview
    2. Click Push
    3. Go to AWS CodeCommit console.
    • Click Repositories
    • Type KinesisProducer to the search bar
    • Click KinesisProducer
    1. You will see the files that you have added into the repository listed in the browser view.

    Clone the repository on the Kinesis Producer EC2 instance

    1. Open Command Prompt and execute the below command
    chmod 400 <path to your key-pair file>
    1. Execute the below command
    ssh -i <path to your key-pair file> ec2-user@<EC2InstanceKinesisProducer>

    Replace <EC2InstanceKinesisProducer> by the value of EC2InstanceKinesisProducer in the Output tab of the DevAx-05 stack

    61. Execute the below command

    git config --global credential.helper '!aws codecommit credential-helper $@'
git config --global credential.UseHttpPath true
git clone https://git-codecommit.<your_region>.amazonaws.com/v1/repos/KinesisProducer

    Replace <your_region> by your Region

    • Type User and Password we saved in step 41 to Username section and Password section
    1. Execute the below command
    cd KinesisProducer
/opt/apache-maven-3.3.9/bin/mvn package shade:shade
    1. It will take a few moments to complete the build. Prequistite
    2. The Producer needs source data, and is expecting a CSV file in the current working folder, called SalesData.csv.
    • Execute the below command to copy this file to EC2 instance
    wget https://workshops.devax.academy/monoliths-to-microservices/module5/files/SalesData.csv

    Prequistite

    Configure ElasticSearch index mappings

    When data is sent to ElasticSearch, it tries to infer the data types in the content, and usually does a good job of getting the data types right. In our case, we are sending the date/time of each of the sales record as long representation (ticks since Epoch) and so ElasticSearch would interpret this field as just a number (of type long). Instead, we need to warn ElasticSearch that it should interpret this timestamp field as a Date. We do this by creating Mappings in the index before we send any records. 65. On the terminal window you have the SSH connection to the Kinesis Consumer EC2 instance, Execute the below command

    curl -X PUT "https://<ElasticsearchEndpoint>/transactions/" -k -d '{
	"mappings": {
		"transaction": {
			"properties": {
				"timestamp": {
					"type": "date"
				},
				"productLine": {
					"type": "string"
				},
				"retailerType": {
					"type": "string"
				},
				"productType": {
					"type": "string"
				},
				"product": {
					"type": "string"
				},
				"quantity": {
					"type": "long"
				},
				"unitCost": {
					"type": "double"
				},
				"totalCost": {
					"type": "double"
				}
			}
		}
	}
}';
    1. If successful, you will see the message {“acknowledged”:true} Prequistite