genailogs - let Googles AI Analyze your log file(s)

JGrana · Apr 2, 2024

@aru created a Python script to send /tmp/syslog.log to Googles Gemini AI for it's analysis.

I used his original script and was somewhat impressed with the results from Google Gemini AI. ~80% pretty good comments, a few, well, somewhat interesting ;-)

The original script was based on python. Knowing that some will not want to install python/pip and the fact I’m not good at all with python, I decided to convert it to /bin/sh (no bash needed) and add some bells and whistles, an install/uninstall/update ala Addons method and a Readme.

A few caveats:

1) I’m no html programmer. I am learning via this script - the Addon web page is pretty primitive. I would like it to have a similar look and feel to things like spdmerlin, etc. - (where’s @Jack Yaz when you need him - LOL). I also would (and am researching) a “Refresh” button for the web page. Unfortunately you have to run “genailogs” from a command line (and maybe cron) to get results updates…

2) Speaking of cron - that’s on my todo list. Add/remove a cron job. Kind of on the fence since the log (or logs) are dynamic and every run of genailogs is a “snapshot” of some recent period of time. I have most of it coded - will release it when it's ready.

3) Some of the bells and whistles (command line option):

noweb - show the question (and log) and results to the terminal (stdout) rather than update the web page. If genailogs is run from stdin, it pipes the output through more. If run from a script or a redirect - it just streams everything (good for scripting).

verbose - show the q/a to the terminal and update the web page.

watch - run genailogs in a continuous loop every SLDELAY seconds (default = 5 minutes) and show on the screen (stdout). Press any key to exit.

4) Genailogs can send multiple log files. The default is just /tmp/syslog.log. If you want others, just add them to the LOGFILES variable in the conf file. I have skynet added to mine.
You can also adjust the number of lines of the log file sent for analysis. The default is 40 on the day genailogs is run. Edit NUMLINES in the conf to adjust this.
The github Readme has more details.

5) Don’t shoot the messenger - the results can be pretty helpful - it discovered 2 devices on my network asking for the same IP address. But, there are times where it’s results are a bit “obtuse”.

6) Genailogs requires a Google Gemini API key. It’s pretty easy, @aru gives good instructions.
Go to the link, get a Key and add it to the .conf file. More information is in the Readme

7) Version 0.4.2 introduced an interactive "chat" mode. You can ask various questions and get responses from GeminiAI. Fun stuff. BTW, If you keep asking the same question - you will often get more/different/ responses.

Here is the help command output:

genailogs (Ver 0.4.2) - send one or more log files (or chat) to Google Gemini AI
and display the results in an Addon web page and/or the terminal

Usage: genailogs [help] [install] [uninstall] [update] [results] [noweb] [verbose] [cron add|del|run] [chat]

help - show this message
results - just show the response/results from Gemini AI
noweb - don't create the web page, just show the results
verbose - both create the web page and show the results
watch - send logs/get results every SLDELAY seconds in a loop
press any key to exit
cron [add|del|run] - cron job - log the last
additional argument: add - add a cron entry
del - delete the cron job
run - run the cron job
chat - start an interactive chat session with GeminiAI
install - install genailogs and create addon dir and config file
uninstall - remove genailogs and its directory and config file
update - check for and optionally update

Ok, if you want to try it out, here is my github page with genailogs. Look over the Readme to install and how to use all the command line options

GitHub - JGrana01/genailogs: Get Gemeni AI to opine on various log files

Get Gemeni AI to opine on various log files. Contribute to JGrana01/genailogs development by creating an account on GitHub.

github.com

ExtremeFiretop · Apr 2, 2024

Do you know how many times my dyslexic blockhead had to re-read: Ge-Nail-og's...
Until I realized it was "Gen-AI-Log's". Man I need to go to sleep... (Yet I never do...)

Jokes aside, this is great stuff, I've used some AI tools on my logs before when troubleshooting some issues with UBIFS on my node some time ago.
As you identified, for some more basic logs and errors it does a great job, others it falls apart pretty quickly... (or fails to give any useful information)
Other than assumptions you could make yourself maybe.

But as LLM's get better and more advanced, you would by extension see better results in this addon, with literally zero work on your part! (and that's what I like about it lol!)

Speaking of LLM's, I quickly checked the Github page, obviously you need a Gemini API for this (duh) as found in the code here: API_KEY="PutYourAPIKeyHere"
My question is: any specific reason why you must use Gemini?

What if you already have an API key for any of the other large language models like Claude, ChatGPT, etc
Just a thought for a feature request some day.

API request through curl for ChatGPT would be something like:

Code:

curl https://api.openai.com/v1/chat/completions   -H "Content-Type: application/json"   -H "Authorization: Bearer $OPENAI_API_KEY"   -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {
        "role": "system",
        "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair."
      },
      {
        "role": "user",
        "content": "Compose a poem that explains the concept of recursion in programming."
      }
    ]
  }'

Tech9 · Apr 2, 2024

ExtremeFiretop said:
Do you know how many times my dyslexic blockhead had to re-read: Ge-Nail-og's...

Make it easier to remember - it's your router's gynecologist.

ExtremeFiretop · Apr 2, 2024

Tech9 said:
Make it easier to remember - it's your router's gynecologist.

Your hurting my poor poor brain.

I go back to scripting now.

JGrana · Apr 2, 2024

ExtremeFiretop said:
Do you know how many times my dyslexic blockhead had to re-read: Ge-Nail-og's...
Until I realized it was "Gen-AI-Log's". Man I need to go to sleep... (Yet I never do...)

Jokes aside, this is great stuff, I've used some AI stuff on my logs before when troubleshooting some issues with UBIFS on my node some time ago.
As you identified, for some more basic logs and errors it does a great job, others it falls apart pretty quickly... (or fails to give any useful information)
Other than assumptions you could make yourself maybe.

But as LLM's get better and more advanced, you would by extension see better results in this addon, with literally zero work on your part! (and that's what I like about it lol!)

Speaking of LLM's, I quickly checked the Github page, obviously you need a Gemini API for this (duh) as found in the code here: API_KEY="PutYourAPIKeyHere"
My question is: any specific reason why you must use Gemini?

What if you already have an API key for any of the other large language models like Claude, ChatGPT, etc
Just a thought for a feature request some day.

API request through curl for ChatGPT would be something like:

Code:

curl https://api.openai.com/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer $OPENAI_API_KEY" -d '{ "model": "gpt-3.5-turbo", "messages": [ { "role": "system", "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair." }, { "role": "user", "content": "Compose a poem that explains the concept of recursion in programming." } ] }'

Hmm, I'll add ChatGPT and see how well it does. Maybe an option Gemini or ChatGPT.

Thanks for the tip!

Viktor Jaep · Apr 2, 2024

Tech9 said:
Make it easier to remember - it's your router's gynecologist.

Dr. Gena I. Logs at your service!

@JGrana @aru ... feel free to use this image to help promote your genailogs script, K? I'm not even going to ask for a cut to the proceeds... it's on me!

ExtremeFiretop · Apr 2, 2024

Viktor Jaep said:
Dr. Gena I. Logs at your service!

View attachment 57644

Now we are talking Viktor!
Break it down for my poor brain, don't lengthen the word.

This picture will get posted all over my walls as a reminder.

JGrana said:
Hmm, I'll add ChatGPT and see how well it does. Maybe an option Gemini or ChatGPT.

Thanks for the tip!

No worries!
I think it's a cool idea, and I'll be following as I actually want to see more "natural" use-cases for such a feature to be used and requested!

JGrana · Apr 4, 2024

I have updated genailogs - available on my github page.

/usr/sbin/curl --retry 3 "https://raw.githubusercontent.com/JGrana01/genailogs/master/genailogs" -o "/jffs/scripts/genailogs" && chmod 0755 /jffs/scripts/genailogs && /jffs/scripts/genailogs install

Changes/Additions:
1) Cron job - genailogs now supports a cron job function. The default time is one min. past every hour (this can be changed in the genailogs.conf file).
Based on the cron job time, genailogs will send a portion of the log(s) to GeminiAI and save it both in a log file (/jffs/addons/genailogs/log/genlogX)
and update the web page. When the web page is viewed, the logs/analysis are time stamped and there is a light blue line separating each log.
The number of log files saved are also configurable in genailogs.conf. Default is 5 logs. The logs are rotated, LIFO.
2) There is now a "Back" button on the web page. - Note - if anyone knows how to execute a shell scripts from the Web page using a button, let me know.
I'm still learning the Asuswrt-Merlin web environment...
3) Added an update command. If there is a new version on github and you run "genailogs update", it will offer to install the newer version.
4) Various bug fixes, mostly in grammar (spelling is not my forte).

I have been looking at OpenAI integration/option. Going is slow - OpenAI complains the files I send it are not in json format... Also, I discovered in lots of testing/debugging - it looks like OpenAI charges for "tokens". Ive already run out and had to add a few $$ to my account. So far, Googles Gemini hasn't limited or charged - and I have done lots of querys.

visortgw · Apr 4, 2024

JGrana said:
I have updated genailogs - available on my github page.

/usr/sbin/curl --retry 3 "https://raw.githubusercontent.com/JGrana01/genailogs/master/genailogs" -o "/jffs/scripts/genailogs" && chmod 0755 /jffs/scripts/genailogs && /jffs/scripts/genailogs install

Changes/Additions:
1) Cron job - genailogs now supports a cron job function. The default time is one min. past every hour (this can be changed in the genailogs.conf file).
Based on the cron job time, genailogs will send a portion of the log(s) to GeminiAI and save it both in a log file (/jffs/addons/genailogs/log/genlogX)
and update the web page. When the web page is viewed, the logs/analysis are time stamped and there is a light blue line separating each log.
The number of log files saved are also configurable in genailogs.conf. Default is 5 logs. The logs are rotated, LIFO.
2) There is now a "Back" button on the web page. - Note - if anyone knows how to execute a shell scripts from the Web page using a button, let me know.
I'm still learning the Asuswrt-Merlin web environment...
3) Added an update command. If there is a new version on github and you run "genailogs update", it will offer to install the newer version.
4) Various bug fixes, mostly in grammar (spelling is not my forte).

I have been looking at OpenAI integration/option. Going is slow - OpenAI complains the files I send it are not in json format... Also, I discovered in lots of testing/debugging - it looks like OpenAI charges for "tokens". Ive already run out and had to add a few $$ to my account. So far, Googles Gemini hasn't limited or charged - and I have done lots of querys.

Will updates continue to overwrite the .conf file? I had to re-enter log file locations and generate/add new API key.

ExtremeFiretop · Apr 4, 2024

JGrana said:
I have been looking at OpenAI integration/option. Going is slow - OpenAI complains the files I send it are not in json format... Also, I discovered in lots of testing/debugging - it looks like OpenAI charges for "tokens". Ive already run out and had to add a few $$ to my account. So far, Googles Gemini hasn't limited or charged - and I have done lots of querys.

This worked for me just now:

Code:

#!/bin/sh

OPENAI_API_KEY='KEY'


# JSON payload
json_payload='{
    "model": "gpt-3.5-turbo",
    "messages": [
      {
        "role": "system",
        "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair."
      },
      {
        "role": "user",
        "content": "Compose a poem that explains the concept of recursion in programming."
      }
    ]
}'


# Use curl to send the request
curl https://api.openai.com/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d "$json_payload"

visortgw · Apr 4, 2024

I believe that there is an issue with multiple log files being parsed:

Code:

# genailogs

Question, Log content, and GenAI analysis result for /jffs/syslog.log saved to /tmp/var/wwwext/user9.asp
grep: /opt/var/log/dnsmasq.log: input file is also the output

Question, Log content, and GenAI analysis result for /opt/var/log/dnsmasq.log saved to /tmp/var/wwwext/user9.asp

My .conf file contains:

Code:

LOGFILES="/jffs/syslog.log /opt/var/log/dnsmasq.log"

alan6854321 · Apr 5, 2024

What's the procedure for getting an API key?

I goto https://ai.google.dev/tutorials/setup
Click the "Get an API key" button

But then I get taken here https://ai.google.dev/available_regions
Which seems to be some kind of error/you're not authorised type page.

(Hmmm .... I notice the UK/Great Britain/England isn't in the list of 'Available regions')

ExtremeFiretop · Apr 5, 2024

alan6854321 said:
(Hmmm .... I notice the UK/Great Britain/England isn't in the list of 'Available regions')

I'm assuming this would be a good use-case for other LLM's as mentioned above.
You would likely need to use any of the other LLM's available in your region or wait until Gemini is available
I'm in Canada by the US and it took us way longer than expected to get Gemini. Lol!

Viktor Jaep · Apr 5, 2024

ExtremeFiretop said:
I'm assuming this would be a good use-case for other LLM's as mentioned above.
You would likely need to use any of the other LLM's available in your region or wait until Gemini is available
I'm in Canada by the US and it took us way longer than expected to get Gemini. Lol!

alan6854321 said:
(Hmmm .... I notice the UK/Great Britain/England isn't in the list of 'Available regions')

Could you not temporarily just create a VPN tunnel that exits in the US, get your API key, see if you can get the script working, get some results, disconnect your VPN and hope your API key/session will continue to work? Who knows... worth a try... but probably not worth having to keep a continues US-based VPN exit up for at all times.

JGrana · Apr 5, 2024

visortgw said:
Will updates continue to overwrite the .conf file? I had to re-enter log file locations and generate/add new API key.

Hmm, shouldn’t. I just download the script. It shouldn’t touch the .conf file.
I will be uploading a new version soon. Let’s see how that goes.

( Sorry about the new API key..)

JGrana · Apr 5, 2024

visortgw said:

I believe that there is an issue with multiple log files being parsed:

Code:

# genailogs

Question, Log content, and GenAI analysis result for /jffs/syslog.log saved to /tmp/var/wwwext/user9.asp
grep: /opt/var/log/dnsmasq.log: input file is also the output

Question, Log content, and GenAI analysis result for /opt/var/log/dnsmasq.log saved to /tmp/var/wwwext/user9.asp

My .conf file contains:

Code:

LOGFILES="/jffs/syslog.log /opt/var/log/dnsmasq.log"

Thanks for the info. What version are you running?

alan6854321 · Apr 5, 2024

Viktor Jaep said:
Could you not temporarily just create a VPN tunnel that exits in the US, get your API key, see if you can get the script working, get some results, disconnect your VPN and hope your API key/session will continue to work?

Tried it using a VPN to Norway - which is on the list.
But got the same result - maybe it knows about common VPN's

visortgw · Apr 5, 2024

JGrana said:
Thanks for the info. What version are you running?

v0.3.1.

alan6854321 · Apr 5, 2024

Viktor Jaep said:
Could you not temporarily just create a VPN tunnel that exits in the US, get your API key, see if you can get the script working, get some results, disconnect your VPN and hope your API key/session will continue to work?

Oh - actually I just tried opening it in an incognito window and it asked me to login to my google account first - so that's how it knows.

I guess I could create a fake account somewhere else in the world, but I'm not sure it'd work from a UK IP address. Too much effort.

JGrana · Apr 5, 2024

visortgw said:

I believe that there is an issue with multiple log files being parsed:

Code:

# genailogs

Question, Log content, and GenAI analysis result for /jffs/syslog.log saved to /tmp/var/wwwext/user9.asp
grep: /opt/var/log/dnsmasq.log: input file is also the output

Question, Log content, and GenAI analysis result for /opt/var/log/dnsmasq.log saved to /tmp/var/wwwext/user9.asp

My .conf file contains:

Code:

LOGFILES="/jffs/syslog.log /opt/var/log/dnsmasq.log"

See if the log file actually exists. I’m feeding both syslog.log and skynet.log and don’t get this error….

Thread starter	Title	Forum	Replies	Date
	Let's talk about Twingate	Asuswrt-Merlin AddOns	11	May 13, 2023
	New Addon - ChatAI - have a chat session with Googles Gemini AI Bot	Asuswrt-Merlin AddOns	2	Apr 9, 2024

genailogs - let Googles AI Analyze your log file(s)

Very Senior Member

Senior Member

Part of the Furniture

Senior Member

Very Senior Member

Part of the Furniture

Senior Member

Very Senior Member

Very Senior Member

Senior Member

Very Senior Member

Senior Member

Senior Member

Part of the Furniture

Very Senior Member

Very Senior Member

Senior Member

Very Senior Member

Senior Member

Very Senior Member

Similar threads

Similar threads

Sign Up For SNBForums Daily Digest