This project builds virtual machine which can be used for analytics of tshark -T ek (ndjson) output. The virtual appliance is built using vagrant, which builds Debian 10 with pre-installed and pre-configured ELK stack.
After the VM is up, the process is simple:
- decoded pcaps (
tshark -T ek output/ ndjson) are sent over
TCP/17570to the VM
- ELK stack in VM will process and index the data
- Kibana is running in VM and can be accessed on
Clone source code
git clone https://github.com/H21lab/tsharkVM.git
Build tshark VM
sudo apt updatesudo apt install tshark virtualbox vagrantbash ./build.sh
Upload pcaps to VM
# copy your pcaps into ./Trace# run following script bash upload_pcaps.sh # or use tshark directly towards 127.0.0.1 17570/tcptshark -r trace.pcapng -x -T ek > /dev/tcp/localhost/17570
Open Kibana with browser
Open Main Dashboard and increase time window to e.g. last 100 years to see there the sample pcaps.
SSH to VM
cd ./VMvagrant ssh
cd ./VMvagrant destroy default
cd ./VMvagrant up
cd ./VMvagrant halt
SSH into VM and check if ELK is running correctly
cd ./VMvagrant sshsudo systemctl status kibana.servicesudo systemctl status elasticsearch.servicesudo systemctl status logstash.service
Elasticsearch mapping template
In the project is included simple Elasticseacrh mapping template generated for the
frame,eth,ip,udp,tcp,dhcp protocols. To handle additional protocols efficiently it can be required to update the mapping template in the following way:
# 1. Create custom mapping, by selecting required protocolstshark -G elastic-mapping --elastic-mapping-filter frame,eth,ip,udp,tcp,dns > ./Kibana/custom_tshark_mapping.json# 2. Deduplicate and post-process the mapping to fit current Elasticsearch versionruby ./Public/process_tshark_mapping_json.rb# 3. Upload file to vagrant VMcd VMvagrant upload ../Kibana/custom_tshark_mapping_deduplicated.json /home/vagrant/tsharkVM/Kibana/custom_tshark_mapping_deduplicated.jsoncd ..# 4. Connect to VM and upload template in the Elasticsearchcd VMvagrant sshcd tsharkVM/Kibanacurl -X PUT "localhost:9200/_index_template/packets_template" -H 'Content-Type: application/json' [email protected]_tshark_mapping_deduplicated.json
Alternative can be using the dynamic mapping. See template
./Kibana/template_tshark_mapping_dynamic.json. And consider setting the numeric_detection parameter true/false depending on the mapping requirements and pcaps used. Upload the template into Elasticsearch in similar way as described above.
tshark -G elastic-mapping --elastic-mapping-filter mapping could be outdated, it is not following properly the Elasticsearch changes and the output can be duplicated. The manual configuration and post-processing of the mapping template is required.
Program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY.
Special thanks to people who helped with the Wireshark development or otherwise contributed to this work:
Example pcap in ./Traces subfolder was downloaded from https://wiki.wireshark.org/SampleCaptures
Created by Martin Kacer
Copyright 2021 H21 lab, All right reserved, https://www.h21lab.com