Use Case: You got a postgres database and you need to move data to elastic search for exploration

Setup Elastic Search on Ubuntu

1. sudo apt update
2. sudo apt install default-jdk
3. wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
4. Add the Elasticsearch repository to the package manager echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-7.x.list
5. Update the package manager sudo apt update
6. Install Elasticsearch:sudo apt install elasticsearch
7. Configure Elasticsearch:
   • sudo vi /etc/elasticsearch/elasticsearch.yml
   • Inside the file, find the network.host setting set it to the IP address of your server or use 0.0.0.0 to listen on all network interfaces.
   • if using network host as 0.0.0.0 set, discovery.seed_hosts: [“127.0.0.1”, “[::1]”]
   • set security –
     • xpack.security.enabled: true
     • xpack.security.transport.ssl.enabled: true
8. Start and enable Elasticsearch:
   • sudo systemctl start elasticsearch
   • sudo systemctl enable elasticsearch
9. Verify Elasticsearch installation: -XGET http://localhost:9200
10. sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive

Setup LogStash

1. Install Logstash: https://www.elastic.co/downloads/logstash
2. Create a Logstash Configuration File: postgresql.conf
3. Run Logstash: bin/logstash -f /path/to/postgresql.conf

Sample postgresql.conf

input {
  jdbc {
    jdbc_connection_string => "jdbc:postgresql://localhost:5432/postgres"
    jdbc_user => "username"
    jdbc_password => "password"
    jdbc_driver_library => "/path/postgresql.jar"
    jdbc_driver_class => "org.postgresql.Driver"
    statement => "select id, name, date_of_birth from employee;"
    jdbc_default_timezone => "UTC"
    jdbc_fetch_size => 1000
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "employee_index"
    document_id => "%{employee_id}"
    user => "elastic"
    password => "password"
  }
}

Interesting read on how Event Prioritization and Introducing a Data Stream to manage data asynchronously helped the team to increase the performance of the system and handle corner cases.

1. They prioritized events to make sure important events do not suffer
2. Introduced a layer (stream) instead of writing directly to the database
3. Reduce Conumse priority if the time taken is beyond a limit
4. Rate Limiting to filter out probable DOS events

https://engineering.razorpay.com/how-razorpays-notification-service-handles-increasing-load-f787623a490f