Impinj® ItemSense™ Troubleshooting Guide

Table of Contents

Introduction

This troubleshooting guide catalogs common issues, organized by symptom, within three categories: setup, jobs, and data collection. For each symptom, the likely causes and the resolutions to each of those causes are described.

If you are already aware of the cause of an issue and are just looking for the steps for fixing it, see the Common Procedures section.

This guide assumes familiarity with Linux commands. See the manual pages for details about individual Linux commands.

Issues

Setup

Installation fails to complete

During the ItemSense installation, the script displays "cannot start service itemsense" or "encountered errors while bringing up the project". The script may continue to run, but it eventually exits with a status 1. The most likely cause is a conflict on one of the ports that ItemSense requires, as specified by the port requirements in the Resource Requirements guide. For example, if an Apache web server is running, it is consuming port 80.

Resolution:

  1. Use the following command to see which port is being consumed:

    sudo netstat -tlnp | grep ':80\W\|:3010\W\|:443\W\|:5672\W'
    
  2. Stop the process that is consuming the port and uninstall the package.

  3. Go to the directory where you extracted the ItemSense archive file. Rerun the ItemSense installation script in the tarfile directory. For specific instructions, follow the steps beginning at step 5 of the Install Instructions in the Installation Guide.

Unable to modify reader NTP settings

With Octane Firmware 5.10 and greater, Network Time Protocol (NTP) settings cannot be modified without first disabling the NTP service.

Resolution:

Use the following command to disable the NTP service:

config network ntp disable

Reader provisioning fails

Networking issues may cause reader provisioning failures. Although ItemSense may be able to communicate with the readers, the readers may not be able to communicate with ItemSense.

Resolution:

  1. Verify connectivity between the readers and ItemSense.
  2. If no networking issues were found, attempt to re-provision the reader.
  3. If you are still unable to provision the reader, contact Impinj support.

ItemSense operates slowly when reporting tags

When the readers are provisioned, the IP address or hostname used to access the ItemSense Management Console (IMC) is written to the reader as the ItemSense address. If this is an IP address or hostname that the reader has trouble connecting to, then the reader will have connectivity issues as it tries to communicate back to the IMC.

Resolution:

  1. When provisioning the readers, be sure to access the IMC using the same IP Address that the readers will be expected to use. This would be the IP address the readers use to communicate back to the IMC.
  2. If the wrong address was used, you may have to re-provision a reader using the correct address.
  3. If the issue persists, contact Impinj support.

Lost administrator password

If the ItemSense administrator password has been lost, it is possible to reset it back to its default value "admindefault".

Resolution:

  1. Connect to ItemSense server using the Secure Shell (SSH) protocol.

  2. Use the following command to edit the admin.json file:

    sudo vi ~/containers/itemsense/home/itemsense/data/configstore/users/admin.json
    
  3. Replace the contents of the file with the following text:

    {"name":"Admin","roles":["Admin"],"visible":true,"passwordHash":"$2a$10$INiYM7LUIq1SDh90qSn6T.tEA1NpIs1/fXhUoOKrQaXGMbKvwlHnu"}
    
  4. Reboot the ItemSense server, and log in to the IMC using "admindefault" as the password.

Reader Health

Reader is in questionable state of health

The overall health status of a reader is either NOT_RESPONDING or IDLE with last check-in reporting null, as shown in the following screenshot. A reader should check in with ItemSense at 15-second intervals. The most likely cause is that the reader is no longer paired with this instance of ItemSense or the reader is disconnected from the network.

Reader health shows status of NOT_RESPONDING or IDLE with last check-in reporting null"

Resolution:

  1. Determine reader and ItemSense pairing. If the reader is indeed paired to this instance of ItemSense, verify connectivity between the reader and ItemSense; otherwise, proceed to step 2.
  2. Note the IP address of the reader in the reader definition. You will need the reader's IP address to re-pair it to your instance of ItemSense.
  3. Delete the reader by clicking on EDIT at the bottom of the reader definition.
  4. Click on the SCANNER tab on the Readers page.
  5. Select the subnet that includes the IP address of the reader that you deleted.
  6. Click the DISCOVER READERS button. After ItemSense discovers the readers, the reader that you deleted should be listed under Available for registration.
  7. Select the reader and click the REGISTER SELECTED READERS button.

Tag Data Collection

Wrong facility is reported on tag reads

There may be several causes of this issue.

Resolution:

Contact Impinj Support.

No tags show up in the Items screen

NTP time-synchronization issues are the most common cause of tag-read issues. Readers must be synchronized with each other and with the ItemSense server. The following situations may prevent this synchronization from happening:

  • Option 42 (DHCP option to use NTP servers) is not enabled in the DHCP client configuration file
  • The DHCP server used by the ItemSense server is different from the DHCP server used by the readers
  • An internet-based NTP server is not accessible and no NTP server is configured on the local network

Network issues or a lack of sufficient disk space may also cause tag-read issues.

Resolution:

  1. Check the job configuration and verify that no filters are preventing tags in the field of view from being read.

  2. Visually inspect the LED lights of the readers to verify that the jobs are running and that they have not been rebooted since the start of the job.

  3. Verify that NTP is working properly:

    ntpq -pn
    

    This command displays information about the peer servers specified in the NTP configuration file. Sample output is shown below. A * or + symbol at the beginning of the line indicates that the peer can be used for synchronization. If no peer is marked by a * or a +, NTP is not operating properly.

    Information displayed when NTP is working properly

    If NTP is working properly, proceed to the next step; otherwise, set up the ItemSense server as an NTP server and configure the readers to use the ItemSense server as their NTP server, as described in the ItemSense Installation document.

  4. Verify connectivity between the readers and ItemSense.

  5. Confirm that there is enough disk space on the ItemSense server. See the Server Requirements section in the ItemSense documentation for resource requirements.

  6. Prune the ItemSense logs to reclaim disk space.

  7. If problems persist, contact Impinj support.

RabbitMQ message queue silently dies after one hour

The RabbitMQ broker will quit after an hour of inactivity (no tag reads), and the connected app will not receive any notification that the connection is broken.

Resolution:

No resolution is necessary — this behavior is by design.

Jobs

Job prematurely stops running or fails to start

If a job stops running before it completes or if a job is unable to start and the IMC displays the following message, the problem may be that there is not enough disk space on the ItemSense server.

Message "Error creating job"

Resolution:

  1. Confirm that there is enough disk space on the ItemSense server. See the Server Requirements section in the ItemSense documentation for resource requirements.
  2. Prune the ItemSense logs to reclaim disk space.
  3. If jobs still do not start, contact Impinj Support.

ItemSense is stopped during a Job

If ItemSense is stopped while a job is running, it should be able to continue to receive data from the reader once it restarts. Data may be lost if reader’s buffer fills before ItemSense is started.

Resolution:

  1. Restart ItemSense to reconnect to the job.
  2. If ItemSense is unable to reconnect, it may be necessary to re-provision the reader.

A reader is stopped, or loses its network connection, during a Job

If the network is temporarily disconnected from a reader while the reader is running a job, ItemSense does not display an error message to indicate that it has lost contact with the reader.

Resolution:

  1. Reconnect the reader and the reads will successfully continue.
  2. If the reader buffer did not overflow, no data was lost.

Common Procedures

Verify connectivity between the readers and ItemSense

Although ItemSense may be able to communicate with the readers, the readers may not be able to communicate with ItemSense. Use the following steps to verify that the readers can communicate with ItemSense.

NOTE: The ports used for reader communication to ItemSense are different from the ports used for ItemSense communication to a reader. For information about the network ports used by ItemSense and the readers, see the Network Requirements section in the ItemSense documentation for resource requirements.

  1. Log on to the ItemSense server.

  2. Use the ifconfig command to see which network interface is being utilized for connectivity between ItemSense and the readers.

  3. Use the tcpdump command to test for reader-to-ItemSense connectivity issues. For example, if eth1 is the network interface, use the tcpdump command as follows:

    • To test for basic reader-to-ItemSense connectivity, specify port 443:

      tcpdump -i eth1 port 443 -w tcpdumpfile.443.pcap
      
    • To test reader-agent provisioning, specify port 51505:

      tcpdump -i eth1 port 51505 -w tcpdumpfile.51505.pcap
      
    • To test SSH commands sent to the reader, specify port 22:

      tcpdump -i eth1 port 22 -w tcpdumpfile.22.pcap
      
    • To test NTP communication from the reader to ItemSense, specify port 123:

      tcpdump -i eth1 port 123 -w tcpdumpfile.123.pcap
      
  4. When the tcpdump command fails or completes, close tcpdump and load the resulting .pcap file into your preferred network traffic analyzer.

Prune the ItemSense logs

ItemSense logs are contained within the home directory of the user who installed it. Each ItemSense service will have its own directory within the ItemSense parent directory. Generally, log files are only created when ItemSense is being used. In other words, no logs are created if ItemSense isn't being used.

  1. Log into the ItemSense server using the ItemSense installer’s login with an SSH or FTP client.

  2. Delete the largest logs from the general ItemSense logs directory.

    File path: ~/containers/itemsense/home/itemsense/var/output/logs

  3. Delete the largest logs from the API service ItemSense logs directory.

    File path: ~/containers/itemsense/home/itemsense/ItemSenseAPIService/var/output/logs

  4. Delete the largest logs from the coordinator ItemSense logs directory.

    File path: ~/containers/itemsense/home/itemsense/Coordinator/var/output/logs

Re-provision a reader

Perform the following steps when a reader needs to be provisioned:

  1. Delete the reader configuration from the IMC if it exists.

  2. Connect to the reader using SSH.

  3. Remove the reader’s CAP file:

    config image removecap
    
  4. Close the SSH client, and use the IMC to re-add the reader.

Determine reader and ItemSense pairing

You can determine the instance of ItemSense with which a reader is paired by examining the reader's config.json file. Use the following steps to retrieve this file from a reader.

  1. Connect to the reader via secure shell:

    ssh root@<reader hostname>
    
  2. Once connected, use the config network RShell command to enable FTP on the reader:

    config network ftp enable
    
  3. Exit out of RShell and connect to the reader via an FTP client:

    ftp <reader hostname or reader IP address>
    
  4. Change to the reader's cust directory, download the config.json file, and exit out of your FTP client. The commands may vary slightly, depending on the FTP client that you use. The following commands are an example:

    cd /
    cd cust
    get config.json
    quit
    
  5. Display the contents of the reader's config.json file by using the cat command:

    cat config.json
    

    Example output:

    {"agentid":"xArray-11-3F-5F","apikey":"94a6546f-7e61-4860-b22a-3cf762e7cf10","baseurl":"https://10.200.10.23:443/itemsense/","tlscertfile":"/cust/cert.pem"}
    

    The contents of the baseurl field in the .json file specify the instance of ItemSense that the reader is paired with.