1

Python server for Wialon Retranslator v1.0

Topic: Python server for Wialon Retranslator v1.0

Estoy recibiendo los datos retransmitidos por una unidad a mi servidor con suse he habilitado un puerto por el cual recibo el paquete.
Existe algún ejemplo para realizar el análisis del mismo, he realizado algunas prueba con Paython


I am receiving data transmitted by a drive to my server with suse I have enabled a port for which I receive the package.
There is an example for the analysis of it, I have done some testing with Paython

2

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

Hello

Good place to start is Python documentation https://docs.python.org/2/library/socket.html#example

3

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

Estimado gracias por responder // Dear thanks for replying

He leído la documentación pero los datos que recibo no se si son correctos ya que no concuerdan con la documentación, te adjunto el código que estoy utilizando para analizar los datos
Lo he mirado y no encuentro el error

//I have read the documentation but the data I receive not know if they are correct and that do not match the documentation, I attached the code I'm using to analyze data
I've looked and can not find the error

gracias

Post's attachments

Attachment icon recibo_binario.py 1.93 kb, 702 downloads since 2015-04-09 

4

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

gabrielmontaldo
Your code is for server
How do you send information to it?

5

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

The code is for the server, not if this good or is there something better
Data is sent: Wialon Retranslator 1.0. Protocol description
The structure's I have, but the truth is I this complicating generate information
Deputy pdf with information

Post's attachments

Attachment icon WialonRetranslator 1.0_en.pdf 70.66 kb, 928 downloads since 2015-04-13 

6

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

shmi

The code is for the server, not if this good or is there something better
Data is sent: Wialon Retranslator 1.0. Protocol description
The structure's I have, but the truth is I this complicating generate information
Deputy pdf with information

7

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

gabrielmontaldo
Now i see

I created python server example, hope this help
https://gist.github.com/ashmigelski/21c … 55444c4f26

PS not tested a lot, check before use

8

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

work nice,  thanks by the example !

--
Spread the AVL- World
9

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

Dear, your contribution impeccable thank you for your help

10

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

Shmi
Estimado, acudiendo a tu ayuda
En el código de ejemplo tengo diferencia en el tamaño ya que hay datos que no muestra bien.
Mi consulta es como analizar o armar el tamaño de acuerdo a la documentación

Dear, attending your help
In the sample code I have difference in size as there is no data that shows well.
My question is how to analyze or arm the size according to the documentation

11

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

Hello gabrielmontaldo
Can you please clarify your question and give some more information about your problem?

12

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

shmi

Saludos, como siempre agradezco tus respuestas

Te cuento, retomando el tema del WialonRetranslator, ejecutando el código que colocaste de ejemplo, no esta analizando bien los datos.
El problema estaría en el tamaño de unos de los parametros.
Soy muy nuevo en esto de paython, he mirado el código y también el documento de WialonRetranslator, no he podido encontrar o darme cuenta del error.
Si me indicas como se analiza el código de ejemplo que colocaste es este mismo post ya que aun no logro identificar lo que realiza el mismo

saludos

Greetings, as always I appreciate your answers

I tell you, back to the issue of WialonRetranslator running the code you placed an example, analyzing the data is not good.
The problem would be in the size of some of the parameters.
I am very new to paython, I looked at the code and document WialonRetranslator, I could not find or realize the error.
If you refer to me as the sample code that you placed this post is analyzed as yet not identify which performs the same

greetings

Post's attachments

Attachment icon WialonRetranslator 1.0_en.pdf 70.66 kb, 270 downloads since 2015-07-22 

13

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

shmi

Coloco el código de ejemplo

Put the sample code

Post's attachments

Attachment icon servertodo.py 4.87 kb, 749 downloads since 2015-07-22 

14

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

gabrielmontaldo wrote:

I tell you, back to the issue of WialonRetranslator running the code you placed an example, analyzing the data is not good.

I didn't tested this code a lot and not all params types parsing implemented, e.g it skips text parameters

        if data_type == 1:
            # text
            # TODO
            pass

Can you post some packages where error occure? Without them i cant find any bug

15

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

shmi

Adjunto paquete de ejemplo

Sample package

Post's attachments

Attachment icon 356612022821377_1.doc 39 kb, 323 downloads since 2015-07-23 

16

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

Its already parsed message data and i cant see any error there. Can you point where problem is?

And raw binary package can be useful for debugging

17

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

shmi

El parámetro marcado de velocidad esta en 0 siempre, aparentemente es un problema en el tamaño del parámetro
Soy nuevo en Paython, no se como obtener el mensaje original, como se analiza el código que tengo de ejemplo aun no lo entiendo, no se como toman los datos finales o como obtener el mensaje original.

The speed dial this parameter to 0 always, apparently is a problem in the parameter size
I am new to Paython, not how to get the original message, as the code example I have yet to understand is analyzed, not as they take the final or obtain the original message data.

18

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

Now i get it and found where bug was, please change following row in parser.py

-                (v['s'], v['c'], v['sc']) = parse('> h h b', block['data_block'], 12)
+               (v['s'], v['c'], v['sc']) = parse('> h h b', block['data_block'], 24)

There was incorrect offset in posinfo parsing. According to protocol

8   Float     Lon – longitude
8   Float     Lat - latitude
8   Float     Height
// 24 bytes offset before next params (was 12 in code)
2   Integer   Speed
2   Integer   Course
1   Integer   Number of satellites

Thanks for reporting!

19

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

Hello shmi,

Pretty awesome parser. Only have the doubt with the long values parsed as Im noticing that for J1708_fl_used=418568 data that Wialon at site show is not being "sent" or parsed correct, the parser gives 'J1708_fl_used': 1829312470712320000 which is too far...

Could be due position parsed?

after  print binascii.hexlify(packet)

383637353231303231353637323635005665314e000000030bbb000000270102706f73696e666f000000006015b756c00000004022632c4000000000004059400010001c0c0bbb0000000f000468646f7000000000000000f03f0bbb0000000e00046f646f0000000000f7454b400bbb00000011010361766c5f696e70757473000000a0d10bbb0000000f00046164633200e3a59bc4204849400bbb000000100004616463313200000000008052ca400bbb0000000a00036d636300000002c00bbb0000000a00036d6e6300000000010bbb0000000a00036c616300000000140bbb0000000e000363656c6c5f6964000000587c0bbb000000090003746100000000000bbb0000000e000367736d5f6c766c00000000460bbb000000120003776865656c5f737065656400000000000bbb0000001000036163635f706564616c00000000000bbb000000150005746f74616c5f6675656c0000000000000003910bbb0000000e000363616e5f666c73000000005b0bbb0000000f000363616e5f7461686f00000000000bbb00000019000563616e5f656e67696e655f6872730000000000000000000bbb00000016000563616e5f6d696c656167650000000000000000000bbb000000120003656e67696e655f74656d7000000000000bbb0000001800054a313730385f656e675f687273000000000000043bf80bbb0000001800054a313730385f666c5f7573656400000000000006633c0bbb0000001300034a313730385f666c5f6c766c000000005b


outcome:
{'params':
    {'acc_pedal': 0, 'total_fuel': 8143352551215988736, 'engine_temp': 0,
        'posinfo': {'c': 350, 'h': 49.0, 'lon': 14.07550048828125, 'sc': 12, 's': 14, 'lat': -90.87934112548828},
        'J1708_eng_hrs': -1712770835237830656, 'gsm_lvl': 71, 'adc2': 51.2694, 'odo': 55.798614501953125, 'mnc': 1, 'can_engine_hrs': 0, 'ta': 0, 'wheel_speed': 0, 'cell_id': 27846, 'hdop': 1.0, 'mcc': 704, 'adc12': 13658.0, 'lac': 16, 'can_taho': 0, 'can_mileage': 0, 'can_fls': 91, 'J1708_fl_used': 1829312470712320000, 'J1708_fl_lvl': 91, 'avl_inputs': 32977}
       
        ,
        'blocks': [
        {'name': 'posinfo', 'data_type': 2, 'data_block': '\x00\x00\x00 G\xb8V\xc0\x00\x00\x00\x00\xa8&,@\x00\x00\x00\x00\x00\x80H@\x00\x0e\x01^\x0c', 'visibility': 1, 'length': 39, 'type': 3003},
        {'name': 'hdop', 'data_type': 4, 'data_block': '\x00\x00\x00\x00\x00\x00\xf0?', 'visibility': 0, 'length': 15, 'type': 3003},
        {'name': 'odo', 'data_type': 4, 'data_block': '\x00\x00\x00\x009\xe6K@', 'visibility': 0, 'length': 14, 'type': 3003},
        {'name': 'avl_inputs', 'data_type': 3, 'data_block': '\x00\x00\x80\xd1', 'visibility': 1, 'length': 17, 'type': 3003},
        {'name': 'adc2', 'data_type': 4, 'data_block': 'm\xc5\xfe\xb2{\xa2I@', 'visibility': 0, 'length': 15, 'type': 3003},
        {'name': 'adc12', 'data_type': 4, 'data_block': '\x00\x00\x00\x00\x00\xad\xca@', 'visibility': 0, 'length': 16, 'type': 3003},
        {'name': 'mcc', 'data_type': 3, 'data_block': '\x00\x00\x02\xc0', 'visibility': 0, 'length': 10, 'type': 3003},
        {'name': 'mnc', 'data_type': 3, 'data_block': '\x00\x00\x00\x01', 'visibility': 0, 'length': 10, 'type': 3003},
        {'name': 'lac', 'data_type': 3, 'data_block': '\x00\x00\x00\x10', 'visibility': 0, 'length': 10, 'type': 3003},
        {'name': 'cell_id', 'data_type': 3, 'data_block': '\x00\x00l\xc6', 'visibility': 0, 'length': 14, 'type': 3003},
        {'name': 'ta', 'data_type': 3, 'data_block': '\x00\x00\x00\x00', 'visibility': 0, 'length': 9, 'type': 3003},
        {'name': 'gsm_lvl', 'data_type': 3, 'data_block': '\x00\x00\x00G', 'visibility': 0, 'length': 14, 'type': 3003},
        {'name': 'wheel_speed', 'data_type': 3, 'data_block': '\x00\x00\x00\x00', 'visibility': 0, 'length': 18, 'type': 3003},
        {'name': 'acc_pedal', 'data_type': 3, 'data_block': '\x00\x00\x00\x00', 'visibility': 0, 'length': 16, 'type': 3003},
        {'name': 'total_fuel', 'data_type': 5, 'data_block': '\x00\x00\x00\x00\x00\x00\x03q', 'visibility': 0, 'length': 21, 'type': 3003},
        {'name': 'can_fls', 'data_type': 3, 'data_block': '\x00\x00\x00[', 'visibility': 0, 'length': 14, 'type': 3003},
        {'name': 'can_taho', 'data_type': 3, 'data_block': '\x00\x00\x00\x00', 'visibility': 0, 'length': 15, 'type': 3003},
        {'name': 'can_engine_hrs', 'data_type': 5, 'data_block': '\x00\x00\x00\x00\x00\x00\x00\x00', 'visibility': 0, 'length': 25, 'type': 3003},
        {'name': 'can_mileage', 'data_type': 5, 'data_block': '\x00\x00\x00\x00\x00\x00\x00\x00', 'visibility': 0, 'length': 22, 'type': 3003},
        {'name': 'engine_temp', 'data_type': 3, 'data_block': '\x00\x00\x00\x00', 'visibility': 0, 'length': 18, 'type': 3003},
        {'name': 'J1708_eng_hrs', 'data_type': 5, 'data_block': '\x00\x00\x00\x00\x00\x04;\xe8', 'visibility': 0, 'length': 24, 'type': 3003},
        {'name': 'J1708_fl_used', 'data_type': 5, 'data_block': '\x00\x00\x00\x00\x00\x06c\x19', 'visibility': 0, 'length': 24, 'type': 3003},
        {'name': 'J1708_fl_lvl', 'data_type': 3, 'data_block': '\x00\x00\x00[', 'visibility': 0, 'length': 19, 'type': 3003}],
    'flags': 3, 'id': '867521021567265', 'time': 1449469468}

20

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

Hi singleroot

Wrong byte order was used. Fixed now

-            v = parse('q', block['data_block'])
+            v = parse('> q', block['data_block'])

Thanks for reporting

21

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

Thanks a lot Shmi!

Just tested and now the values are parsed correctly.

22

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

Hello guys. Any ideea how can i save all the incoming data into a valid json file?
ANd another thing, i am trying to insert into mysql db some incoming params like adc, mileage...etc.
In parser.py i have entered this:


def insert(unu, doi):
    cursor = db.cursor()
    sql = "INSERT INTO data (imei, adc) VALUES ('%s', '%s')" % (unu, doi)
    try:
        cursor.execute(sql)
        db.commit()
    except:
        # Rollback in case there is any error
        #db.rollback()

def parsePacket(packet):
...
if name == "adc12":
            avl = str(v)
             insert(msg['id'], avl)
...
        

     
It works, but it inserts the data multiple times in one execution, so i have many duplicated rows each time the script executes on incoming data.
For example:
867521021626384|12618.0|2016-04-21 05:30:51
867521021626384|12618.0|2016-04-21 05:30:51
As you can see, the same values at the same time.

Any idee why?.
Thanks!

23

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

Hi florin1
Parsed messages are Python dicts and can be dumped to JSON. The only thing - I recomend to delete 'blocks' with raw data before it
Regarding your second question - you  can try to save messages after parsePacket return parsed message.

Here's some hints

import json
...
msg = parsePacket(packet)
msg.pop('blocks', None)

// 1st question
print json.dumps(msg)

// 2nd question
value = msg['params'].get('adc12', None)
if value is not None:
    insert(msg['id'], value)
...
24

Python server for Wialon Retranslator v1.0

(edited by florin1 30/04/2016 20:49:28)

Re: Python server for Wialon Retranslator v1.0

Thanks shmi.
I have another question. I have made modifications to the code, and added support for multiple connections. So the params i need are stored into mysql db. It works, but, in comparison with the wialon platform, the messages i receive have a big delay. So for example, while on wialon platform i see the real time position of a unit, on my app i see it with delay, somehow in the past. It is like the whole messages coming from the retranslators, are kept in a big buffer, waiting to be written. My question is, does this have anything to do with the "queue" part of the code and how can i modify it, so that i can reduce this delay? And another problem is that many messages don't have posinfo {lat, long}....

# -*- coding: utf-8 -*-

import binascii
import socket
import MySQLdb
from parser import parse, parsePacket
import threading
from datetime import datetime
import json


def insert(unu, doi, mil, lat, lon, fuel): # unu = imei, doi = adc
    db = MySQLdb.connect(host="localhost",
                     user="root",    
                     passwd="xxxxx",
                     db="dbdb") 
                    

    cursor = db.cursor()  
    read = "SELECT COUNT(1) FROM data WHERE imei = '%s'" % (unu)
    cursor.execute(read)
    if cursor.fetchone()[0]:
        print "Imei existent: " + unu + ". Doar update."
        update = "UPDATE data SET adc = '%s', mileage = '%s', fuel = '%s', latitudine = '%s', longitudine = '%s', created = '%s' WHERE imei = '%s'" % (doi, mil, fuel, lat, lon, datetime.now(), unu)
        try:
            cursor.execute(update)
            db.commit()
        except MySQLdb.Error, e:
            print "MySQL Error %d: %s" % (e.args[0], e.args[1])

    else:
        
        sql = "INSERT INTO data (imei, adc, mileage, latitudine, longitudine, created) VALUES ('%s', %s, '%s', '%s', '%s', '%s')" % (unu, doi, mil, lat, lon, datetime.now())
        try:
            cursor.execute(sql)
            db.commit()
        except MySQLdb.Error as e:
            # Rollback in case there is any error
            #db.rollback()
            print "MySQL Error %d: %s" % (e.args[0], e.args[1])

    cursor.close()
    del cursor
    db.close() 

CONNECTION = (socket.gethostname(), 6054)

server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# prevent 'ERROR: Address already in use'
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)


server.bind((socket.gethostname(), 6054))
server.listen(15)


def clientthread(conn):
     queue = ''
     while True:
     #Receiving from client
         data = conn.recv(1024) # 1024 stands for bytes of data to be received
         if not data:
             break
         # append to queue
         queue = queue + data
         
         
         # get first packet size
         packet_size = parse('<i', queue)
         if packet_size + 4 <= len(queue):
             # get packet
             packet = queue[4:packet_size + 4]
             # print binascii.hexlify(packet)
        
             msg = parsePacket(packet)
             msg.pop('blocks', None)
             pwrb = msg['params'].get('adc12', None) # BCE
             pwrt = msg['params'].get('pwr_ext', None) #Teltonika
             fuelb = msg['params'].get('fuel level', None) # BCE
             if pwrb is not None:
                 adc = pwrb
             if pwrt is not None:
                 adc = pwrt
             if fuelb is not None:
                 fuel = fuelb
             elif fuelb is None:
                 fuel = "None"
            
             mil = msg['params'].get('mileage', None)
             posinfo = msg['params'].get('posinfo', None)
             if posinfo is not None:
                 lat = msg['params']['posinfo'].get('lat', None)
                 lon = msg['params']['posinfo'].get('lon', None)
                 
                 print "Imei: " + str(msg['id']) + ". Pozitie: " + str(lat) + " | " + str(lon)
                 insert(msg['id'], adc, mil, lat, lon, fuel)
             else:
                 print ("Position Info is invalid.")
                 
            
             # remove packet from queue
             queue = queue[packet_size + 4:]
             
             # packet was received successfully
             conn.send(str(0x11))
             

while True:
    #Accepting incoming connections
    print "Server pornit {0} on {1}".format(*CONNECTION)
    # Accept connections
    sck, addr = server.accept()
    print "Conectare de la {0}:{1}".format(*addr)
    t = threading.Thread(target=clientthread, args=(sck,))
    t.daemon = True
    t.start()
    
    
   
conn.close()
sck.close()
25

Python server for Wialon Retranslator v1.0

Re: Python server for Wialon Retranslator v1.0

Ok, i don't know if it is the right approach, but it seems to work. So basically, now i have multiple connections support and writing to mysql in realtime. For writing all the data in queue at once, i have added the for statement:

queue = queue + data
         
         for i in range(0,len(queue)):
             # get first packet size
             packet_size = parse('<i', queue)
             if packet_size + 4 <= len(queue):
                 # get packet
                 packet = queue[4:packet_size + 4]
                 # print binascii.hexlify(packet)
            
                 msg = parsePacket(packet)
                 ...

and in parser.py:

def parse(fmt, binary, offset=0):
    '''
    Unpack the string

    fmt      @see https://docs.python.org/2/library/struct.html#format-strings
    value    value to be formated
    offset   offset in bytes from begining
    '''
    parsed = 0
    try:
        parsed = struct.unpack_from(fmt, binary, offset)
        parsed = parsed[0] if len(parsed) == 1 else parsed
    except:
        pass
    
    return parsed