Final Post for Project One

It has been over a month since I first met Tessa, and I am glad to say that we have both learned from each other! On her end, she learned how to press the bar and receive reinforcements for multiple fixed ratio schedules. She also learned to trust me as her handler and caregiver. On my end, I learned how to handle and treat a trainee in order to successfully train the behavior. I also learned more about the various learning topics (magazine training, shaping, schedules, variability of behavior, etc.) discussed in lecture.

I thoroughly enjoyed the various interactions I had with Tessa throughout the training process. I would handle her each time I came to feed her, and I would typically play with her after completing a training session. We grew fond of each other through the process of training, and I was proud of her for each milestone she accomplished. The most exciting part of training was observing the learning process, which only took around two weeks to complete.

One of the more difficult facets of the training process was dealing with her climbing behaviors. I thought it was humorous when she would try to climb out of the box, but that humor soon turned to frustration when it became evident that the behavior was impacting the efficiency of her training. Although I was able to mitigate the behavior, she was easily frustrated when exposed to a new schedule and quick to satiate. The amount of food given beforehand did not seem to affect how quickly she would become satiated, so I assume that she was quickly bored by the training process.

Tessa climbing on the front of the box - she climbed everywhere!

I was glad to interact with Tessa on such a close level, and I feel that I have developed a close bond with an organism I once viewed as disgusting. I have a newfound respect for rats as a species, and I have developed a better understanding of their role in scientific research. I am excited to continue training Tessa for my next project!

Sniffy and Tessa: A Comparison of Virtual and Live Rat Training

Shaping Sniffy the Virtual Rat was definitely easier than shaping Tessa. This was due to the lower variability in the program's behaviors compared to the live rat. Although the program did include a variety of behaviors, it was simple to understand what procedures were having the best effect on shaping by observing the bar-sound graph's progress. For example, reinforcing Sniffy for rearing behaviors almost always resulted in him approaching the bar. It took nearly one hour to shape Sniffy to press the bar. In contrast, the shaping process for Tessa was a lot more involved than with the virtual rat. It was necessary for me to be much more alert because I was attempting to reward behaviors such as looking towards the bar and moving towards the bar. These behaviors were present in the virtual rat, but they were difficult to detect because of graphical limitations. Tessa was more variable in her behaviors, which was beneficial for shaping because I was able to quickly determine which behaviors could lead to bar-pressing. However, her tendency to rear and jump towards the top of the cage was a problem, especially when she jumped on the bar and received reinforcement for jumping behavior. Shaping Tessa took a total of two days.

Magazine training was similar for both Sniffy and Tessa. By rewarding both organisms immediately after they received reinforcement, I was able to keep them both near the food magazine until they began associating the magazine with reinforcement. Tessa was more likely than Sniffy to respond to delivered reinforcements when she was on the opposite side of the box. This allowed me to encourage Tessa to return to the magazine, and I was then able to quickly deliver reinforcements. Magazine training for Tessa was successful within the first training session (approximately after 15 minutes), and it took a total of 19 minutes to magazine train Sniffy.

Figure 1. Cumulative record for FR-5 schedule of reinforcement for Tessa.

It was very useful to have both a virtual and live rat during this training process. The virtual rat prepared me to interact with the live rat by providing an understanding of the process of magazine training and shaping. Understanding the importance of paying close attention to the organism was emphasized in the Sniffy program - whenever I missed a behavior that should be reinforced, the graph of bar-sound association would decrease. Therefore, I was more likely to pay attention to my live rat when shaping her. The program also helped me form an understanding of the different schedules of reinforcements. Due to time constraints, I was only able to complete a FR20 schedule with Tessa. However, I was able to complete a VR50 schedule with Sniffy over the course of an hour. I recommend using Sniffy in future classes, and possibly requiring students to complete multiple schedules of reinforcement to understand how the behavior varies between schedules.

All in all, I found it very useful to train the virtual rat before training my live rat. Even though I paid more attention to my live rat, the virtual training process contributed to my understanding of the training process. Tessa's high variability of behavior did affect the training process, but I did not run into any major hurdles because I understood the underlying processes involved in training the bar-pressing behavior.

Variable Ratio Training and Extinction of Sniffy the Virtual Rat

Goal:
To train Sniffy the Rat to press the bar on a VR-50 schedule of reinforcement. A variable ratio schedule of reinforcement may reward the behavior for different amounts of responses, but the average number of responses required to receive reinforcement is 50. After Sniffy was performing successfully on the VR-50 schedule, extinction was performed where Sniffy did not receive any reinforcement for bar-pressing behavior.

Procedure:
Sniffy was moved up to a VR-5 schedule of reinforcement following shaping. The ratio was subsequently stretched through VR-10, VR-20, VR-35, and finally VR-50. Sniffy was isolated during each schedule, allowing the program to quickly learn the behavior on its own. Each schedule was considered learned when bar-sound and action strength graphs reached their maximum. Once the VR-50 schedule was learned, Sniffy was exposed to extinction until the graphs representing bar-sound's association with reinforcement and action strength fell to zero.

Results:
Sniffy successfully adapted to each VR schedule and made it to VR-50 without any issues. The entire process took around 30 minutes in real-time, but the program was isolated to run at a high rate. Each schedule was continued until the program recorded a cumulative record indicative of the appropriate VR schedule. Extinction was successful and Sniffy no longer associated bar-pressing and sound with reinforcement. The action strength of Sniffy's response also fell to zero, indicating that the response was unlikely to occur in the future without additional shaping.

Discussion:
The training of Sniffy on the VR schedule was relatively simple because I was able to isolate the experiment and let Sniffy learn on his own. I did not return to the program until the bar-sound and action strength graphs reached their maximum. I ensured that Sniffy was performing the bar-pressing behavior reliably before stretching the ratio. Sniffy was less likely to explore the box during the variable ratio schedules. I assume this is because Sniffy understood that bar-pressing would yield a reinforcement, but he was not aware of how many bar-presses were required. Therefore, he was likely to remain at the bar to receive reinforcement. On average, the acquisition of a new VR schedule took around 30 minutes in virtual time. Sniffy slowly responded during the beginning of each schedule, but his responses became more consistent as he learned the schedule.

Cumulative record of VR-50 schedule indicating the schedule had been learned by the program.

Extinction of Sniffy was also a very simple task. He pressed the bar repeatedly during the beginning of extinction, but his response rate dropped sharply near the end of extinction. Extinction was considered to be complete when Sniffy was no longer performing the instrumental response.

Cumulative record of extinction. Note that Sniffy reaches a response rate of zero.

Figure 1. Graph of bar-sound and action strength for VR-50 (left) vs. Extinction (Right).



Visual: Average Responses per Training Session

The average numbers of responses recorded during each 30-minute training session are recorded below. Shaping, FR3, FR10, FR15, FR20, and Extinction are averaged between two days of training. Additional statistics on training sessions can be found here.

Schedule	Average Responses
Magazine Training	28
Shaping (1 and 2)	61
FR1	124
FR2	200
FR3 (1 and 2)	219
FR5	429
FR10	451
FR15 (1 and 2)	810
FR20 (1 and 2)	788
Extinction (1 and 2)	352

Food Deprivation Results

Tessa was deprived of food for the duration of training in order to motivate her to seek reinforcement (sucrose pellets) in the Operant box. Tessa's target weight was 203 grams. Adjustments were made to maintain a weight near this target. Tessa was not fed on Saturday, October 6th. Therefore, she was fed twice on Sunday, October 7th.

Date	Weight (g)	Food (g)
12-Sep	233	2.9
13-Sep	220	3.6
14-Sep	213	5.1
15-Sep	208	4.1
16-Sep	203	10
17-Sep	206	6
18-Sep	201	7.7
19-Sep	200	8.4
20-Sep	201	8
21-Sep	205	5
22-Sep	205	4.6
23-Sep	199	6.5
24-Sep	200	7
25-Sep	201	6.2
26-Sep	201	6
27-Sep	198	7
28-Sep	203	6.6
29-Sep	203	6.3
30-Sep	200	7.5
1-Oct	199	7.7
2-Oct	202	7.3
4-Oct	198	8.4
5-Oct	202	7.6
6-Oct	N/A	0
7-Oct	192	14.2
8-Oct	200	Free Feed

Battling Tessa the Ninja-Rat: Reducing Unwanted Behavior

During Tessa's first day on the FR2 schedule, she became extremely frustrated with the bar-pressing task and attempted to escape from the box. She investigated each side of the cage for any way to escape. For instance, she tried pressing her way through the top of the front window of the Operant box. When she returned to the bar, she realized that there was a second light above the magazine which she could grasp with her front paws. She began to rear, then suddenly she jumped from the floor of the Operant box and grabbed onto the light with her front paws. Tessa pulled herself to the very top of the box, and with all of her might she kicked her hindlegs to the light fixture and grasped onto it with her feet. Being the most agile of her sisters, she was quick to kick herself from the light fixture to the middle of the top of the Operant box. She grasped onto the top of the box and attemped to escape from the top of the box (as shown below).

Even though she was unable to pull herself through this small opening, it did not deter her from trying to escape every day during training. Many sessions would end with Tessa receiving a reinforcement, using her ninja-rat skills to attempt to escape, failing and falling from the top of the Operant box, then returning for reinforcement and repeating the cycle. I thought this behavior was affecting the efficiency of her training, so I attempted to correct it for FR15 and FR20 schedules.

Goal:
To reduce the amount of time Tessa spent attempting to escape the Operant box through the top of the box by blocking the opening in the top of the box.

Procedure:
In order to completely block Tessa from escaping through the top of the box, I attained a large clipboard and placed it over the top of the box. This blocked the large opening in the top of box as well as smaller openings she could grasp onto towards the front of the box.

Results:
Tessa spent less time attempting to escape through the top of the Operant box, but she continued to explore the box at a comparable rate. She spent more time trying to push her face through the bottom of the box.

Discussion:
Tessa's variable behavior was very useful in shaping the bar-pressing behavior because it gave me a lot of possible behaviors to attempt to train towards bar-pressing. However, her excitability did pose a problem during training sessions because she would spend a great deal of time exploring the box when she could be training. The clipboard was successful in preventing her from escaping from the top of the box, but it did not stop her from climbing to the top of the box. I think it would have been more useful to train an incompatible behavior (such as rewarding her for staying on the bottom of the cage), but the time restraints on this project made the restriction of escape a more viable option.

I believe this adjustment did make a difference in the final days of training, and she spent more time training during the FR15 and FR20 schedules than exploring the box. However, this change could be attributed to the fact that she had to work harder to receive reinforcement than in previous schedules. Likewise, she was less likely to become satiated because reinforcements took longer and more work to receive. Nevertheless, preventing her from receiving the satisfaction of putting her head outside of the box prevented her from attempting to escape from the top of the box during the final days of training and extinction.

On a more general note, I believe my training could have been improved by extending the amount of time spent working in the Operant box. Because shaping lasted for an average of 20 minutes, I believe Tessa expected future training sessions to only last around 20 minutes. She performed a majority of her bar-presses during the first ten minutes of training sessions, and typically she was satiated by the time 15 minutes had been elapsed. However, Tessa did respond during the final minutes of training sessions after 20 or 25 minutes of training. Therefore, I believe Tessa made adequate progress in her training regardless of her tendency to respond at dramatically lower rates towards the end of each training session. She learned the amount of bar-presses required to receive reinforcement, and I never ended a training session until she was consistently responding on the desired schedule.

Training Sessions 14 and 15: Extinction

Goal:
To eliminate the trained bar-pressing behavior through the use of extinction. Extinction works by restricting reinforcement of a previously reinforced behavior until the organism no longer performs the behavior.

Procedure:
In order to eliminate the behavior of bar-pressing for reward, Tessa was no longer reinforced for performing an bar-pressing. Bar presses did not result in reinforcement, the mechanical sound of the magazine, nor the flickering of the light above the bar in the Operant box. Extinction was performed for thirty minutes on two sequential dates.

Results:
Extinction Day One: Tessa performed 165 bar-presses during the first five minutes of the extinction session. This far exceeds the previous amount of 20 bar-presses required to receive reinforcement. Table 1 contains the running number of bar-presses recorded during the training session. Figure 1 represents the amount of bar-presses performed during each five-minute period.

 Table 1

Time Elapsed (minutes)	Total Bar Presses Performed
5	165
10	297
15	383
20	436
25	511
30	519

Figure 1. This graph represent the extinction of the bar-pressing behavior. The amount of bar-presses per five-minute period is graphed against total time elapsed.

Extinction Day Two: Tessa performed 90 bar-presses during the first five minutes of this extinction session. This is much less than the amount of bar-presses performed on Extinction Day 1, indicating that extinction did have an effect on her response rate. Table 2 contains the running total of bar-presses recorded during the training session. Figure 2 illustrates the amount of responses performed during each five-minute period.

Table 2

Time Elapsed (minutes)	Total Bar Presses Performed
5	90
10	126
15	145
20	152
25	179
30	184

Figure 2. This graph represent the extinction of the bar-pressing behavior. The amount of bar-presses per five-minute period is graphed against total time elapsed.

Discussion:

Extinction Day One:

Cumulative record for Extinction Day 1

The effects of extinction on Tessa's bar-pressing behavior are evidenced by the change in response rates within and between extinction sessions. During the first day of extinction, Tessa pressed the bar continuously for the first two minutes before realizing no reinforcement would be given. The fact that she pressed the bar a total of 165 times in the first five minutes may represent an Extinction Burst. When she did not receive reinforcement for the first 20 responses, she became frustrated and pressed the bar over eight times as much as required in the FR20 schedule. She then ceased responding for two minutes, then responded continuously for two minutes, followed by another break from responding for two minutes. Towards the end of the session she went four minutes without responding. These trends indicate that Tessa's bar-pressing behavior was becoming extinct.

Extinction Day Two:

Cumulative record for Extinction Day 2

During the second day of extinction, Tessa pressed the bar in a more intermittent fashion. She would press the bar for less than thirty seconds before taking a break. The fact that she pressed the bar a total of 90 times during the first five minutes of extinction day 2, compared to a total of eight times during the last five minutes of extinction day 1, may represent Spontaneous Recovery of the bar-pressing behavior. Although she should have expected no reward from the behavior, she continued to press the bar at a high rate when reintroduced to the experimental environment. Towards the end of the session, she did not press the bar for a span of about seven minutes. This indicates that the bar-pressing behavior had been extinguished sufficiently.

Training Sessions Five through Thirteen: Fixed Ratio Schedules of Reinforcement

Goal: 
To successfully train Tessa to press the bar for an increasing number of required behaviors. In order to reach the final behavior requirement of 20 bar-presses per reinforcement (FR 20), I gradually stretched the ratio from the original FR1 through FR2, FR3, FR5, FR7, FR10, and FR15.

Procedure: 
For each fixed ratio level, Tessa was reinforced for performing the correct number of bar-presses dictated by the FR level. Tessa was not manually reinforced for any behaviors, nor was she encouraged to approach the bar through additional reinforcement. Each session was run for a minimum of twenty minutes. Runtimes for each session are below:

Date	Schedule	Runtime (minutes)
9/25	FR2	23.75
9/26	FR3	21.58
9/27	FR3	22.00
9/28	FR5	21.50
9/29	FR7	22.67
9/30	FR10	28.52
10/01	FR15	21.77
10/02	FR15	24.00
10/04	FR20	29.33
10/05	FR20	22.26

 

Results:
Tessa performed a minimum of 200 bar-presses (FR2) and a maximum of 810 bar-presses (FR10) during her Fixed Ratio Training. FR3 and FR15 schedules were performed over multiple days to ensure Tessa learned the new schedule. As shown in Figure 1, Tessa performed more bar-presses on day 2 of the FR3 and day 2 of the FR15 schedules.

Figure 1: Total number of instrumental behaviors (bar-presses) per training session. Refer to Table 1 for session runtimes. Sessions are listed in the order they were performed.

Discussion:
Tessa made considerable progress through the Fixed Ratio Schedules of Reinforcement during her training. Initially, she became frustrated as the requirements for reinforcement were increased. For example, she began to exhibit escapist behaviors during the first day on FR2 schedules. These behaviors included climbing to the top of the Operant box on all sides, as well as actually climbing to the middle of the top of the Operant box. Although these behaviors were not eliminated, they were mitigated by blocking all exits from the box. You can read about how I reduced these behaviors here.

Although I was able to mitigate some of these troublesome behaviors, Tessa's wide variety of behaviors did affect the efficiency of her training. She was likely to perform a multitude of behaviors between each reinforcement, especially after at least ten minutes of training had elapsed. She would typically receive a reinforcement, then spend at least one minute exploring the rest of the Operant box. She would sniff the front window of the box, climb on all of the walls, and put her nose through any openings in the box. Nevertheless, she developed a better understanding of my expectations as training progressed. She was less likely to spend time exploring as the schedules were increased, which may be simply due to the higher requirements of each schedule. Furthermore, Tessa was likely to become distracted by any activity in the laboratory. This was worst when multiple students were in the lab at one time. In order to alleviate this, I used the rat privacy cover for FR10, FR15, and FR20 schedules.

On average, Tessa became satiated during each training session around 20 minutes of training. This lack of instrumental responses could be attributed to the fact that I ended her earliest shaping sessions after running less than 25 minutes of training. She may have expected to be released from the Operant box, and most importantly she may have assumed she would be released at a specific time and received rat chow in the holding room. The amount of rat chow given on the previous day did not seem to affect her motivation to perform in the Operant box. Likewise, I had difficult maintaining her target weight of 203 grams - however, she did not seem to be any less motivated when she gained weight compared to when she lost weight. You can read more about the schedule of food deprivation here.

Tessa's style of bar-pressing was not the desired style: instead of using her paws to press the bar, she typically pressed the bar with her nose or cheeks while holding a paw on the bar. This shows that the shaping of putting her paws on the bar was shaped successfully, but the behavior of pushing on the bar with her paws may have been overlooked - or possibly replaced by the reinforcement of pressing the bar with her cheeks during the FR schedules. Nevertheless, Dr. Trench once quoted B.F. Skinner as saying "a bar-press is a bar-press is a bar-press," so I did not attempt to change Tessa's style of bar-pressing.

The increase in Tessa's bar-pressing behavior was a function of the schedule she was on at the time: when she was on a higher schedule, she pressed the bar more times than on lower schedules. However, she did not achieve fifty reinforcements for each schedule of reinforcement. I thought fifty reinforcements would be a good goal, but she did not achieve this goal in either of the FR20 sessions. She received a maximum of 90 reinforcements (FR2) and an average of 68 reinforcements across all trials. Figure 2 illustrates the number of reinforcements received during each training session.

Figure 2. Total number of reinforcements per training session. Refer to Table 1 for training runtimes. Schedules are ordered by date they were performed.