DrayTek UK Users' Community Forum
Help, Advice and Solutions from DrayTek Users
For splunk users
- robingb
- Topic Author
- Offline
- Junior Member
-
Less
More
- Posts: 10
- Thank you received: 0
20 Aug 2009 20:35 #1
by robingb
For splunk users was created by robingb
I though I would start a thread on the use of splunk as it is very complicated for new users but also very powerfull, so well worth it in the long run.
On my setup I have a Vigor 2820Vn outputting logging data to a NAS running syslog-ng. I run splunk on my PC. I load the previous days log files into splunk to analyse the reports, checking for hack attempts and or compromises in network security.
When you first install splunk it starts logging everything the local host (PC)is doing. If you don't want this data, stop splunk from doing so else it slows down searches. To stop splunk logging PC data go to "Manager" > "Data Inputs" and disable everything in there. There are many layers so it takes 5-10 minutes. By the time you have finished splunk will already have many thousands of entries.
To clear all these entries (and so speed up searches on your router data) go to the command prompt and...
cd c:\program files\splunk\bin
Then stop splunk, e.g. "splunk stop"
Then clear eventdata e.g. "splunk clean eventdata"
Then start splunk again, e.g. "splunk start"
Now you have a clean non logging installation (splunk need an automated way for this!)
Now add your syslog files as a data source, e.g. go to "Manager" > "Data Inputs" > "Files & Directories" > "New" > "Upload a local file" and then select the syslog-ng log file(s) and set host to say "2820", leave source type as "automatic" and set index to "main".
Now you have added your Draytek log data to splunk you can go to the "search" app and search the data. However to really see the benefit of splunk you need to define fields in the data.
When in "search" and looking at some results there is a small white down arrow in a small gray box next to each log entry, click on this and select "Extract Fields".
You can now define a regex for fields, I use the following..
(?i) Vigor: (?P<FIELDNAME>(\[\w+\])|([^:-]*)) = Draytek_Entry_Type
(?i)\[FILTER\]\[(?P<FIELDNAME>[^\]]*)\] = Draytek_Filter_Action
(?i)\[FILTER\]\[\w+\]\[(?P<FIELDNAME>[^,]*), = Draytek_Filter_Direction
(?i)Magic Number: (?P<Draytek_Magic_Number>.*?)\s+# = Draytek_Magic_Number
(?i) UpSpeed=(?P<FIELDNAME>[^ ]*)[ ] = Draytek_UpSpeed
(?i) DownSpeed=(?P<FIELDNAME>[^ ]*)[ ] = Draytek_DownSpeed
(?i) SNR=(?P<FIELDNAME>[^ ]*)[ ] = Draytek_SNR
(?i) Atten=(?P<FIELDNAME>[^ ]*)[ ] = Draytek_Attenuation
(?i) States=(?P<FIELDNAME>[^ ]*)[ ] = Draytek_States
(?i)\[Mode=(?P<FIELDNAME>[^ ]*)[ ] = Draytek_Mode
(?i) inquire (?P<FIELDNAME>.*) = Draytek_DNS_Inquire_Address
(?i) DNS \-> (?P<FIELDNAME>[^ ]*)[ ] = Draytek_DNS_Server_For_Inquire
(?i) User: (?P<FIELDNAME>[^: ]*)[: ] = Draytek_Local_User_IP
(?i)[^a-zA-Z]-[> |>](?P<FIELDNAME>[^:]*): = Draytek_Destination_IP
Hope that helps
On my setup I have a Vigor 2820Vn outputting logging data to a NAS running syslog-ng. I run splunk on my PC. I load the previous days log files into splunk to analyse the reports, checking for hack attempts and or compromises in network security.
When you first install splunk it starts logging everything the local host (PC)is doing. If you don't want this data, stop splunk from doing so else it slows down searches. To stop splunk logging PC data go to "Manager" > "Data Inputs" and disable everything in there. There are many layers so it takes 5-10 minutes. By the time you have finished splunk will already have many thousands of entries.
To clear all these entries (and so speed up searches on your router data) go to the command prompt and...
cd c:\program files\splunk\bin
Then stop splunk, e.g. "splunk stop"
Then clear eventdata e.g. "splunk clean eventdata"
Then start splunk again, e.g. "splunk start"
Now you have a clean non logging installation (splunk need an automated way for this!)
Now add your syslog files as a data source, e.g. go to "Manager" > "Data Inputs" > "Files & Directories" > "New" > "Upload a local file" and then select the syslog-ng log file(s) and set host to say "2820", leave source type as "automatic" and set index to "main".
Now you have added your Draytek log data to splunk you can go to the "search" app and search the data. However to really see the benefit of splunk you need to define fields in the data.
When in "search" and looking at some results there is a small white down arrow in a small gray box next to each log entry, click on this and select "Extract Fields".
You can now define a regex for fields, I use the following..
(?i) Vigor: (?P<FIELDNAME>(\[\w+\])|([^:-]*)) = Draytek_Entry_Type
(?i)\[FILTER\]\[(?P<FIELDNAME>[^\]]*)\] = Draytek_Filter_Action
(?i)\[FILTER\]\[\w+\]\[(?P<FIELDNAME>[^,]*), = Draytek_Filter_Direction
(?i)Magic Number: (?P<Draytek_Magic_Number>.*?)\s+# = Draytek_Magic_Number
(?i) UpSpeed=(?P<FIELDNAME>[^ ]*)[ ] = Draytek_UpSpeed
(?i) DownSpeed=(?P<FIELDNAME>[^ ]*)[ ] = Draytek_DownSpeed
(?i) SNR=(?P<FIELDNAME>[^ ]*)[ ] = Draytek_SNR
(?i) Atten=(?P<FIELDNAME>[^ ]*)[ ] = Draytek_Attenuation
(?i) States=(?P<FIELDNAME>[^ ]*)[ ] = Draytek_States
(?i)\[Mode=(?P<FIELDNAME>[^ ]*)[ ] = Draytek_Mode
(?i) inquire (?P<FIELDNAME>.*) = Draytek_DNS_Inquire_Address
(?i) DNS \-> (?P<FIELDNAME>[^ ]*)[ ] = Draytek_DNS_Server_For_Inquire
(?i) User: (?P<FIELDNAME>[^: ]*)[: ] = Draytek_Local_User_IP
(?i)[^a-zA-Z]-[> |>](?P<FIELDNAME>[^:]*): = Draytek_Destination_IP
Hope that helps
Please Log in or Create an account to join the conversation.
- thrain
- Offline
- Junior Member
-
Less
More
- Posts: 59
- Thank you received: 0
21 Aug 2009 12:46 #2
by thrain
Replied by thrain on topic For splunk users
Splunk is the mutts nuts.
It makes is so easy to locate the data and graph it.
If you can run the syslog server & splunk on the same box.
Just one word of warning, some versions of syslog report the following if the syslog message is repeated :-
Nov 21 04:42:34 router last message repeated 68 times
Nov 21 04:43:35 router last message repeated 153 times
Nov 21 04:44:31 router last message repeated 143 times
Nov 21 04:45:33 router last message repeated 133 times
There is an option to switch the last message repeated off, but I can't remember what it is.
It makes is so easy to locate the data and graph it.
If you can run the syslog server & splunk on the same box.
Just one word of warning, some versions of syslog report the following if the syslog message is repeated :-
Nov 21 04:42:34 router last message repeated 68 times
Nov 21 04:43:35 router last message repeated 153 times
Nov 21 04:44:31 router last message repeated 143 times
Nov 21 04:45:33 router last message repeated 133 times
There is an option to switch the last message repeated off, but I can't remember what it is.
Please Log in or Create an account to join the conversation.
Moderators: Sami
Copyright © 2024 DrayTek