Use Case: You got a postgres database and you need to move data to elastic search for exploration

Setup Elastic Search on Ubuntu

1. sudo apt update
2. sudo apt install default-jdk
3. wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
4. Add the Elasticsearch repository to the package manager echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-7.x.list
5. Update the package manager sudo apt update
6. Install Elasticsearch:sudo apt install elasticsearch
7. Configure Elasticsearch:
   • sudo vi /etc/elasticsearch/elasticsearch.yml
   • Inside the file, find the network.host setting set it to the IP address of your server or use 0.0.0.0 to listen on all network interfaces.
   • if using network host as 0.0.0.0 set, discovery.seed_hosts: [“127.0.0.1”, “[::1]”]
   • set security –
     • xpack.security.enabled: true
     • xpack.security.transport.ssl.enabled: true
8. Start and enable Elasticsearch:
   • sudo systemctl start elasticsearch
   • sudo systemctl enable elasticsearch
9. Verify Elasticsearch installation: -XGET http://localhost:9200
10. sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive

Setup LogStash

1. Install Logstash: https://www.elastic.co/downloads/logstash
2. Create a Logstash Configuration File: postgresql.conf
3. Run Logstash: bin/logstash -f /path/to/postgresql.conf

Sample postgresql.conf

input {
  jdbc {
    jdbc_connection_string => "jdbc:postgresql://localhost:5432/postgres"
    jdbc_user => "username"
    jdbc_password => "password"
    jdbc_driver_library => "/path/postgresql.jar"
    jdbc_driver_class => "org.postgresql.Driver"
    statement => "select id, name, date_of_birth from employee;"
    jdbc_default_timezone => "UTC"
    jdbc_fetch_size => 1000
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "employee_index"
    document_id => "%{employee_id}"
    user => "elastic"
    password => "password"
  }
}