Birdfeeder app notes, technical, etc

edro · October 23, 2021, 2:33pm

I’ve updated to the latest Vizy software (0.80) and tried running the birdfeeder app. It aborts right away with the excerption below in the Vizy console window. It’s looking for a bird feeder_consts.py file?

2021-10-16 11:23:10.017358: E tensorflow/core/platform/hadoop/hadoop_file_system.cc:132] HadoopFileSystem load error: libhdfs.so: cannot open shared object file: No such file or directory
cp: cannot stat ‘/home/pi/vizy/apps/birdfeeder/birdfeeder_consts.py’: No such file or directory

Unhandled exception:
Traceback (most recent call last):
File “/usr/local/lib/python3.7/dist-packages/vizy-0.0.80-py3.7.egg/vizy/importconfig.py”, line 18, in import_config
return __import(basename, etc_filename, imports)
File “/usr/local/lib/python3.7/dist-packages/vizy-0.0.80-py3.7.egg/vizy/importconfig.py”, line 7, in __import
spec.loader.exec_module(module)
File “”, line 724, in exec_module
File “”, line 859, in get_code
File “”, line 916, in get_data
FileNotFoundError: [Errno 2] No such file or directory: ‘/home/pi/vizy/etc/birdfeeder_consts.py’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/pi/vizy/apps/birdfeeder/main.py”, line 276, in
bf = Birdfeeder()
File “/home/pi/vizy/apps/birdfeeder/main.py”, line 43, in init
self.config_consts = import_config(consts_filename, self.kapp.etcdir, [“THRESHOLDS”, “PESTS”, “ALBUM”, “DEFEND_BIT”])
File “/usr/local/lib/python3.7/dist-packages/vizy-0.0.80-py3.7.egg/vizy/importconfig.py”, line 22, in import_config
return __import(basename, etc_filename, imports)
File “/usr/local/lib/python3.7/dist-packages/vizy-0.0.80-py3.7.egg/vizy/importconfig.py”, line 7, in __import
spec.loader.exec_module(module)
File “”, line 724, in exec_module
File “”, line 859, in get_code
File “”, line 916, in get_data
FileNotFoundError: [Errno 2] No such file or directory: ‘/home/pi/vizy/etc/birdfeeder_consts.py’

Process has exited.

plord12 · October 16, 2021, 4:06pm

I see the same -

2021-10-16 17:04:28.809354: E tensorflow/core/platform/hadoop/hadoop_file_system.cc:132] HadoopFileSystem load error: libhdfs.so: cannot open shared object file: No such file or directory
cp: cannot stat '/home/pi/vizy/apps/birdfeeder/birdfeeder_consts.py': No such file or directory

Unhandled exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/vizy-0.0.80-py3.7.egg/vizy/importconfig.py", line 18, in import_config
    return __import(basename, etc_filename, imports)
  File "/usr/local/lib/python3.7/dist-packages/vizy-0.0.80-py3.7.egg/vizy/importconfig.py", line 7, in __import
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 724, in exec_module
  File "<frozen importlib._bootstrap_external>", line 859, in get_code
  File "<frozen importlib._bootstrap_external>", line 916, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/home/pi/vizy/etc/birdfeeder_consts.py'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/vizy/apps/birdfeeder/main.py", line 276, in <module>
    bf = Birdfeeder()
  File "/home/pi/vizy/apps/birdfeeder/main.py", line 43, in __init__
    self.config_consts = import_config(consts_filename, self.kapp.etcdir, ["THRESHOLDS", "PESTS", "ALBUM", "DEFEND_BIT"])
  File "/usr/local/lib/python3.7/dist-packages/vizy-0.0.80-py3.7.egg/vizy/importconfig.py", line 22, in import_config
    return __import(basename, etc_filename, imports)
  File "/usr/local/lib/python3.7/dist-packages/vizy-0.0.80-py3.7.egg/vizy/importconfig.py", line 7, in __import
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 724, in exec_module
  File "<frozen importlib._bootstrap_external>", line 859, in get_code
  File "<frozen importlib._bootstrap_external>", line 916, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/home/pi/vizy/etc/birdfeeder_consts.py'

edge · October 16, 2021, 5:11pm

Apparently our QA process has some holes that need plugging… Hold on while we get this sorted out.

Edward

edge · October 16, 2021, 5:42pm

I have been informed that the issue should be fixed. Please give build 81 a try

fingers crossed…

Edward

edro · October 16, 2021, 6:01pm

Yes, working! It starts now and detected a photo of a bluejay.

plord12 · October 16, 2021, 6:07pm

Good for me as well

plord12 · October 17, 2021, 6:20pm

After a day of use here in the UK I’ve still not got a picture that is accurate. Either birds are missed or it thinks a tree is a bird.

Still, I’ve been suffering from sun glare so I’ll keep trying.

But, I see the pre-learnt birds are US … we’ll need to figure out a way to get a bigger UK data set.

edge · October 17, 2021, 6:24pm

Hello, false positives do happen and they often look ridiculous. You can try adjusting the sensitivity if they happen too often. Or adjust the consts file if it’s a particular species/location. It’s true that the birds species in the TensorFlow model are North American. We will offer European species eventually…

An important thing to consider is that we trained the model using birds that took up about 25% of the image – so fairly large, as if you are looking at a bird feeder. This way the pictures get some decent detail of the birds. If you’re far away and birds are small (for example 10% of the image or less) it will not detect (at all, unfortunately.) The state of the art for these kinds of things isn’t perfect… you make trade-offs and assumptions, but improvements are coming along at a fast pace for this kind of deep-learning detection.

Edward

plord12 · October 19, 2021, 5:49pm

I see the model is checked in here - https://github.com/charmedlabs/kritter/tree/main/src/kritter/tf/birdfeeder.

Can you describe how this was generated ?

I guess it should be possible to repeat the training and then do the same with a European data set.

Thanks.

edge · October 20, 2021, 12:12am

Hello,
It’s absolutely possible to train a network on European species. The training for North American species was done using the TensorFlow training utilities. Having heard it described, I don’t think it’s something that can be done in an afternoon, unless you were sufficiently familiar with TensorFlow.

Edward

plord12 · October 20, 2021, 2:10pm

Understood … still it would be great to share where the model came from and references to test data.

BTW one of my goals here is to take the birdfeeder as an example and create a more generic garden wildlife alert camera. First step will be to be able to build, train and deploy myself the existing app … then I can iterate.

edge · October 20, 2021, 2:47pm

Hello,
The training set was obtained from the iNat competition. We combined recent years’ data and trained a network by extracting pictures of the target species. These repositories are an amazing resource We did the training using an AWS server with several CUDA GPUs. You can do this yourself without a fancy GPU, but it can take a long time.

There are several howto guides on how to train your own Tensorflow network. This is a rapidly changing space and if the guide is over a year old, it’s probably already dated. We need to update our models to take advantage of the latest developments… this will happen over the coming months.

Edward

plord12 · October 21, 2021, 8:53am

Many thanks for the image source link

plord12 · October 23, 2021, 1:15pm

Well, I gave this a shot based on https://www.tensorflow.org/tutorials/images/classification and created a trained model from UK birds.

However thinking this through some more, I don’t think this is the right way since the model would need to output bounding boxes as well right ?

Did you do the training with a python script ? If so, can you share ?

Thanks.

edge · October 23, 2021, 2:31pm

Hello,
I believe you need object detection instead of image classification. Here are the latest object detection models. You start with a pretrained classifier (e.g. coco) and retrain it using their script. Since we trained the birdfeeder classifier using mobilenet_v2, there have been several new models added to the mobile models that would be worthwhile giving a try. The birdfeeder has a unique accuracy vs speed requirement. A tensorflow lite model would be too inaccurate, while a resnet model would be too slow(?)

Fortunately, the iNat database has pictures with classifications and bounding boxes. Many of the pictures in the database are not good for training though, for example, a bird on a power line taken from 100 meters.

The original code that we used is on an AWS server somewhere. I’m putting it on my todo list to try to dig up. We will be doing some training fairly soon (weeks?) using special GPU hardware that we purchased. We’ll provide notes/howto guide when this happens, but happy to answer any questions in the meantime. It’s definitely an exercise for the less faint of heart, but very doable.

Edward

plord12 · October 30, 2021, 2:48pm

So … I’ve started following https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html which I believe is the technique you used.

The first issue I hit is that the iNaturalist images at https://github.com/visipedia/inat_comp/tree/master/2021 doesn’t contain bounding box data ( older sets do for some birds … but I didn’t find anything for the UK bird set ). So I’m slowly going through and creating bounding boxes.

So far, the training I’ve tried on my imac (with GPU) hasn’t gone that well … the total loss rises to large values and after a few hours the training seems to stop making progress.

Capturing the steps here https://github.com/plord12/vizy-training FYI.

plord12 · November 1, 2021, 2:55pm

Another issue I’ve found is the current tensorflow version, 2.4.3, doesn’t support the frozen inference graph that vizy is using … ie vizy is using the older tensorflow v1 release.

Will vizy be moving to v2 ?

edge · November 1, 2021, 8:43pm

Hello,
We will be digging into this over the coming months. I’m not sure what version of Tensorflow we’ll end up using, but it will certainly be a more advanced version than what’s installed now. We will target a version of Tensorflow that works best on the Raspberry Pi4 (best speed, accuracy, and reliability) whatever version that is.

Are you interested in using Tensorflow Lite? Please share what you’re trying to accomplish so we can feed this info into our effort.

Edward

plord12 · November 1, 2021, 9:28pm

Ah, okay thanks.

No idea … previously you mentioned that it was too inaccurate.

Well, firstly I thought I would reproduce how you trained the birdfeeder. Then move on to extending it with UK birds and animals so its useful to me in my garden. Tracking at https://github.com/plord12/vizy-training.

plord12 · November 21, 2021, 5:12pm

A little update.

After creating bounding boxes for my dataset ( 10 most common UK birds + Red Kite + Heron + Hedgehog + Fox + Squirrel + Cat ) I tried -

centernet_mobilenetv2_fpn_od - gave errors
ssd_mobilenet_v2_320x320_coco17_tpu-8 - training didn’t converge
but ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8 did actually train okay

With the model I passed through test images and it looked reasonably okay … I’m sure it could do better with more cpu cycles, but a start I think.

Then I tried to save the model as a frozen graph with https://github.com/tensorflow/tensorflow/issues/46740#issuecomment-827609165. After copying to vizy ( replace the birdfeeder frozen_inference_graph.pb/labelmap.pbtxt files I only got

Unhandled exception:
Traceback (most recent call last):
  File "/home/pi/vizy/apps/birdfeeder/main.py", line 351, in <module>
    bf = Birdfeeder()
  File "/home/pi/vizy/apps/birdfeeder/main.py", line 176, in __init__
    self.tflow.open()
  File "/usr/local/lib/python3.7/dist-packages/kritter/tf/tfdetector.py", line 46, in open
    self.image_tensor = detection_graph.get_tensor_by_name('image_tensor:0')
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py", line 3644, in get_tensor_by_name
    return self.as_graph_element(name, allow_tensor=True, allow_operation=False)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py", line 3468, in as_graph_element
    return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py", line 3510, in _as_graph_element_locked
    "graph." % (repr(name), repr(op_name)))
KeyError: "The name 'image_tensor:0' refers to a Tensor which does not exist. The operation, 'image_tensor', does not exist in the graph."