A good chatbot database that can scale

  • Thread starter Thread starter kolleamm
  • Start date Start date
  • Tags Tags
    Database Scale
Click For Summary

Discussion Overview

The discussion revolves around the feasibility of creating an intelligent chatbot capable of learning and conversing with users, focusing on the challenges of data retrieval and processing speed in current hardware. Participants explore the implications of natural language processing, memory management, and the efficiency of existing chatbot technologies.

Discussion Character

  • Exploratory
  • Technical explanation
  • Debate/contested

Main Points Raised

  • One participant expresses concern about the time it may take for a chatbot to find data in memory, suggesting that current hardware may not support intelligent chatbots effectively.
  • Another participant mentions that natural language processing platforms have existed for decades and that existing chatbots like Alexa and Siri are examples of successful implementations, though they are proprietary.
  • There is a suggestion that proper database selection could address concerns about finding the closest match for responses.
  • A participant points out that modern computers often utilize parallel processing and multitasking, which may alleviate some concerns about searching data sequentially.
  • One participant shares their experience with a smart speaker, noting its speed in providing answers, implying that effective chatbots are indeed possible.
  • Another participant challenges the notion that computers search through items one by one, suggesting the use of more efficient data structures like binary search trees.
  • There is a discussion about the hardware requirements for chatbots, with one participant clarifying that purchasing a device like an Alexa only provides the interface, while the heavy processing occurs in the cloud.

Areas of Agreement / Disagreement

Participants express differing views on the practicality of creating intelligent chatbots with current hardware. While some argue that it is feasible and examples exist, others raise concerns about the limitations of data retrieval and processing speed.

Contextual Notes

Participants reference various technologies and programming languages relevant to natural language processing, indicating a range of approaches and tools available for chatbot development. There is also mention of the potential complexity involved in generating natural-sounding speech and managing conversations.

kolleamm
Messages
476
Reaction score
44
Lately I've been really interested in creating a chatbot that could learn and talk to users.
The only problem though that's preventing me from starting is the time it may take for the program to find the data it needs in memory.

Most chatbots simply find the closest match to your words and then reply with the associated reply.

The problem is however finding that closest match may be difficult especially if there is a lot of data.

From what I understand computers search through items one by one, while a single neuron is connected to tens of thousands of other neurons that respond nearly instantly.

So is an intelligent chatbot even practical with today's hardware?
 
Technology news on Phys.org
kolleamm said:
Lately I've been really interested in creating a chatbot that could learn and talk to users.
The only problem though that's preventing me from starting is the time it may take for the program to find the data it needs in memory.

Most chatbots simply find the closest match to your words and then reply with the associated reply.

The problem is however finding that closest match may be difficult especially if there is a lot of data.

From what I understand computers search through items one by one, while a single neuron is connected to tens of thousands of other neurons that respond nearly instantly.

So is an intelligent chatbot even practical with today's hardware?

The OP is in luck as computer platforms capable of natural language processing were developed at SRI speech technology lab and other sites decades ago. Chatbot's are ubiquitous in e-commerce though the well known systems like Alexa & Siri are proprietary. A programmer could buy a chatbox and emulate features within all legal constraints on a generic computer.

Generating natural sounding speech, not to mention "conversations", can be resource intensive often utilizing neural networks and machine learning but your requirements could be met by proper database selection. As you suggest locate closest match and "respond".

FYI two different threads might help. A thread "Computer Programming Languages" mentions several high-level languages apropos to natural language processing but I am not conversant with the packages. A math thread on random string generation and related topics might help depending on what you wish to accomplish.

https://www.physicsforums.com/threa...gramming-languages.912188/page-4#post-6102627
https://www.physicsforums.com/threads/probability-that-a-random-string-is-a-word.961582

Ask about neurons and perception in the Biology Science forum. IMO speech recognition in the human mind has little to do with individual neurons. Searching the topic "neural networks" might help compare human and machine learning. Modern computers usually contain some parallel processing abilities along with "multi-tasking". Only poorly designed code searches "one by one".

On the hobby scale your concern about "finding data in memory" may be unwarranted. Chat by nature is I/O intensive, generating lots of interrupts. Table look ups (TLU) should not require prohibitive cycles compared to the strength of your search algorithms.

Your last question about practical 'bots has already been answered, no? Examples abound. Devising metrics to evaluate NLP performance sounds interesting. Time (cycles) concerns you; consider timing your bot or Alexa or even a speech translator.
 
Last edited:
  • Like
Likes   Reactions: kolleamm
My Axexa smart speaker is pretty fast. Ask her a question and the answer comes rapidly. Presumably, Amazon needs to search the whole internet to find the answer. I think there are simple chat apps that converse.

So it is not only possible, but you can buy one for only $30.See also Eliza.
 
  • Like
Likes   Reactions: kolleamm and Klystron
kolleamm said:
From what I understand computers search through items one by one
No, google 'binary search tree'.

anorlunda said:
So it is not only possible, but you can buy one for only $30.
Well that only gets you the hardware to interface to the Alexa engine which sits in the cloud and does all the hard work. Anyone can build "skills" which can sit within this platform - see https://developer.amazon.com/alexa-voice-service/design
Edit: this is probably a better start point https://developer.amazon.com/alexa-skills-kit
 
pbuk said:
Well that only gets you the hardware to interface to the Alexa engine

The "Alexa engine" being a tens or hundreds of billion dollar computer system.

BoB
 

Similar threads

Requesting suggestions for languages, libraries, and architectures for parallel (and sometimes non parallel) numerical and scientific computations
  • · Replies 8 ·
Replies
8
Views
5K
April Fool's Physics Papers 2026
  • · Replies 2 ·
Replies
2
Views
609
Omnigenetic model for complex traits
  • · Replies 2 ·
Replies
2
Views
4K
Can someone help me pick a technical elective
  • · Replies 8 ·
Replies
8
Views
3K
Undergrad What Key Issues Should Cosmologists Address in the ΛCDM Model?
  • · Replies 1 ·
Replies
1
Views
3K
Undergrad Are researchers really this interested in 'Planet 9'
  • · Replies 6 ·
Replies
6
Views
2K
Graduate Focus all of your ire on me, if you would
  • · Replies 4 ·
Replies
4
Views
3K
The Scientist as Hero - or not
  • · Replies 1 ·
Replies
1
Views
4K
Introducing the PF Library
  • · Replies 183 ·
7
Replies
183
Views
81K
Graduate Black holes in the globular cluster M22 with Dr. Tom Maccarone
  • · Replies 12 ·
Replies
12
Views
4K