A good chatbot database that can scale

[kolleamm]

Lately I've been really interested in creating a chatbot that could learn and talk to users.
The only problem though that's preventing me from starting is the time it may take for the program to find the data it needs in memory.

Most chatbots simply find the closest match to your words and then reply with the associated reply.

The problem is however finding that closest match may be difficult especially if there is a lot of data.

From what I understand computers search through items one by one, while a single neuron is connected to tens of thousands of other neurons that respond nearly instantly.

So is an intelligent chatbot even practical with today's hardware?

 

[Klystron]

Lately I've been really interested in creating a chatbot that could learn and talk to users.
The only problem though that's preventing me from starting is the time it may take for the program to find the data it needs in memory.

Most chatbots simply find the closest match to your words and then reply with the associated reply.

The problem is however finding that closest match may be difficult especially if there is a lot of data.

From what I understand computers search through items one by one, while a single neuron is connected to tens of thousands of other neurons that respond nearly instantly.

So is an intelligent chatbot even practical with today's hardware?

The OP is in luck as computer platforms capable of natural language processing were developed at SRI speech technology lab and other sites decades ago. Chatbot's are ubiquitous in e-commerce though the well known systems like Alexa & Siri are proprietary. A programmer could buy a chatbox and emulate features within all legal constraints on a generic computer.

Generating natural sounding speech, not to mention "conversations", can be resource intensive often utilizing neural networks and machine learning but your requirements could be met by proper database selection. As you suggest locate closest match and "respond".

FYI two different threads might help. A thread "Computer Programming Languages" mentions several high-level languages apropos to natural language processing but I am not conversant with the packages. A math thread on random string generation and related topics might help depending on what you wish to accomplish.

https://www.physicsforums.com/threa...gramming-languages.912188/page-4#post-6102627
https://www.physicsforums.com/threads/probability-that-a-random-string-is-a-word.961582

Ask about neurons and perception in the Biology Science forum. IMO speech recognition in the human mind has little to do with individual neurons. Searching the topic "neural networks" might help compare human and machine learning. Modern computers usually contain some parallel processing abilities along with "multi-tasking". Only poorly designed code searches "one by one".

On the hobby scale your concern about "finding data in memory" may be unwarranted. Chat by nature is I/O intensive, generating lots of interrupts. Table look ups (TLU) should not require prohibitive cycles compared to the strength of your search algorithms.

Your last question about practical 'bots has already been answered, no? Examples abound. Devising metrics to evaluate NLP performance sounds interesting. Time (cycles) concerns you; consider timing your bot or Alexa or even a speech translator.

 

[anorlunda]

My Axexa smart speaker is pretty fast. Ask her a question and the answer comes rapidly. Presumably, Amazon needs to search the whole internet to find the answer. I think there are simple chat apps that converse.

So it is not only possible, but you can buy one for only $30.See also Eliza.

 

[pbuk]

From what I understand computers search through items one by one

No, google 'binary search tree'.

So it is not only possible, but you can buy one for only $30.

Well that only gets you the hardware to interface to the Alexa engine which sits in the cloud and does all the hard work. Anyone can build "skills" which can sit within this platform - see https://developer.amazon.com/alexa-voice-service/design
Edit: this is probably a better start point https://developer.amazon.com/alexa-skills-kit

 

[rbelli1]

Well that only gets you the hardware to interface to the Alexa engine

The "Alexa engine" being a tens or hundreds of billion dollar computer system.

BoB