devmorgan.com

I got an e-mail this morning from a site that linked to me.  I got the 99th slot for “100 Best Blogs for Career-Minded Students“. Kinda cool I guess. I jumped to the site and read the little blurb about me.

Heh. Almost completely wrong.

A student at the Rochester Institute of Technology, this blogger is a computer science major with aspirations of being a programmer.

Not a big deal. In fact, I won’t even tell them to correct it, but I am not a computer science major (I am an Information Technology major). I am also an actual programmer. I do have a job, and I code things.

At least they got the RIT part right ;).

However, this e-mail kinda caused me to revisit a lot of things with this blog and with my life. So, lets start with the tagline to the blog…

Code Monkey and Grad Student Extraordinaire

I still like to consider myself a code monkey. So, thats still fine. Also, technically I am still a grad student… but that is only because I haven’t finished my capstone yet, but I am done with my coursework (although they do let me teach every once and a while). So that’s cool, but there is now a lot more.

I now work as a Lead Integration Analyst for the University of Rochester Medical Center (PS: Nothing I say on this blog reflects their opinions, or any other legal mumbo jumbo). This is a new direction for my career - a step away from web development, and a step towards back end programming.

Another major change is now I am a home-owner. What a crazy change! I spend my spare time fixing up the house or cleaning or changing things. It’s a very different process than what I’ve been used to.

Of course, the most major of changes coming up (not here yet!) is that I am a father. In the next few weeks we will have a newborn and that will take up most of our time. I am excited, and so is Jess. It’s going to be one hell of an experience.

So, I guess what it really comes down to is, I am going through a career change. I am no longer “just a student”, I am a Father, Programmer, Academic, and I guess: Adult.

coding, education, life, rambling, technology

So, before I sit back and talk about all of the actual methods on the MagicDB object, we need to talk about a problem that is specific to PHP and the mysqli extension.

Now, the basic idea I had was to build something that could generate prepared statements dynamically. This will (if done correctly) help protect against SQL injection, and ideally help speed up a few frequently run queries (as long as the SQL structure is always consistent). We need to select mysqli because it has the mysqli_stmt::prepare ( string $query ) function.

Ok, so, now we come to the binding parts of the process… and here is where it gets sticky. We are presented with these two methods:

mysqli_stmt::bind_param ( string $types , mixed &$var1 [, mixed &$... ] )
mysqli_stmt::bind_result ( mixed &$var1 [, mixed &$... ] )

This presents us with a problem. Notice how both methods (which are required for prepared statements) allow for multiple parameters. This is handy — wait.. really.. it’s never handy. I really don’t know why they designed it this way. In fact.. Damn those PHP people. OK.. so maybe I do know why,  but it gets in to reference/value discussion.. so.. lets move on and just figure out a way to fix it.

In an ideal world, for being dynamic, the functions should be designed something more like this:

mysqli_stmt::bind_param ( string $types ,  array &$params )
mysqli_stmt::bind_result (  array &$results )

Instantly this becomes a lot better. Since the parameters/results are stored as an array, they can be dynamically generated and of any length.  To address this concern, we need to tackle both functions a little bit differently. First off, the bind_param function:

I wrote a simple private function that is included in MagicDB that will accept a mysqli_stmt object, and a list of parameters. From it, the function will call the bind_param method with the proper values and return the statement object.

Now, lets discuss the major concern with this code. The data types of the parameter array that is passed in are used for truth. So, as a developer you need to be careful. This will not cause any security risks, but it may reject your parameter if the wrong data type is used. This is a known concern, but one that I am not worried about since I know to pass an int when I need an int. The beauty of this code is in the last couple lines. I use the php variable-variable-name functionality to generate a dynamic variable reference. I call each variable $bind#, and then assign the parameter to it. I then can use the & symbol to add the variable by reference to the array. Using  the PHP function call_user_func_array, we can get around the multiple parameter problem by simple passing an object reference, a method name, and an array of parameters.

Now for the binding of the result. This code isn’t stored in a function, although it could be. It’s stored in my “get” method on the MagicDB object. Some variables are out of scope, but here is the snip.

So here, without getting into too much detail about how the get() function works, we want to return an object of the type that $object is.  The object should have all the values populated with the results of the database. We bind the variables in the same manner that we bound the parameters in the previous function.  Where it gets fancy is in the use of reflection. We get the type of the object, and pass it into what is called a ReflectionClass.

This will generate a generic class of the type we specify. We must then instantiate the object by calling newInstanceArgs() method. This allows us to pass an array with the parameters for the constructor. This is why the constructor is required in all of the data objects. We do this for each iteration of the fetch, and return the array of objects.

Sexy eh?

Next up we talk about the API and how the MagicDB functions actually work.

coding, data, technology

MagicDB is the codename for the Dynamic Data Layer that I built to help me with my much larger project, my fabled “Capstone” for my Masters of Science Degree. First a couple technical details:

• MagicDB currently runs on PHP 5.x.
• MagicDB currently only interfaces with MySQL.
• MagicDB is built using an Object-Oriented paradigm.
• MagicDB uses PHP’s mysqli engine.

Understanding the technical bits will explain why some decisions have to be made. PHP is used pragmatically for myself, since that is the language I used to build my capstone with. It’s free. That helps. MySQL is also free, and one of the most common companions with PHP. The mysqli engine is used primarily because it provides the support for prepared statements. Using a prepared statement allows us to curb SQL injection attacks, which addresses one of our main concerns with security.

MagicDB has only a couple rules that need to be followed.

1. Data Objects (PHP) must follow specific naming requirements.
2. Data Objects (PHP) and Data Tables (MySQL) must follow specific structural requirements.

Naming Requirements are actually pretty simple. All data objects used with MagicDB need to follow the pattern [Prefix]TableName. The prefix can be whatever the developer wants. For my capstone project (codename: TWiG), I used the prefix “TWiGdb_”. This means for a table in my database named “Entry”, I have a corresponding data object with the class name of TWiGdb_Entry.

Structural Requirements are a little more in-depth. Originally, the structures were required to be identical. This meant that for every column in the database, the data object should have a column for that as well. I improved upon this by adding a column to every database table (but not the data object) called “exist”. This column is a flag to show if the row was removed from the database, it has a default value of true. This allows us to “remove” data, without deleting it (although functionality is built to actually DELETE data), in case we require that data later.

Another structural requirement is that the first column in the data object MUST be the primary key for the table. In the table, the first column must be the primary key as well. This allows us to issue UPDATE commands easily. A constructor with all the attributes as parameters is also required. Order of the parameters and attributes must also match across the board.

The final requirement is that each attribute of the data object MUST be using the same exact name as the corresponding column in the table.

For example the table User would be described like this:

The corresponding data object would look like this:
Once we can build the structure of the database, and the corresponding data objects, we can begin to discuss how the DyDL works. MagicDB itself really has a simple interface to use.

In the next few posts we’ll discuss each of these methods, and talk about some of the private methods that power the implementation.

coding, data, technology

So, I figure it’s about time I actually talk a little bit about code. I know many of my readers aren’t programmers, so my apologies. For my capstone work I have been playing around a lot with data, and I want to share my experiences.

So, first things first. Lets talk about Dynamic Database Layers. We want to build a database layer that is as dynamic as possible and flexible, but provides understandable structure and expected results. We want something that is secure, but also usable. So lets break it down:

What are the goals?

1. Ease of use.
2. Low learning curve.
3. Clear and understandable “rules”
4. Secure
5. Solves routine problems naturally.
6. Allows non-routine problems to be solved easily.

Ease of use. The first and foremost goal we have for our DyDL is that its easily usable. Ideally, it should make accessing the data simple and trivial. If you need to write lines upon lines of code just to invoke simple calls, developers won’t use it. Make easy things easy. If a developer needs to do something more complicated, try to make it easy - but they’ll be willing to do more work to accomplish it.

Low learning curve. One of the most difficult tasks as a programmer is to adapt to new technology. Understanding the lingo, the structure, and the benefits is required to be efficient. Don’t make it difficult to use. Use common paradigms to make entry into use easy. This leads us into the next requirement..

Clear and understandable “rules”. Make how the software layer works straight forward and simple. Don’t build an application that has situations where if you want to do one thing, you have to do it differently than another thing. Make it clear to the developer what each function is actually doing. If all data objects require certain structure, make it clear. Giving meaningful messages and documentation for the layer.

Secure. This is straight forward. Make your best attempt to make the code secure. If you can prevent SQL injections, do so. Build as much structure to the framework such that you can limit the ways to exploit it. Don’t trust unknown data.

Solves routine problems naturally. Make simple stuff simple. If you need to do a simple access of data based on a few parameters, make it so that is one simple call. Try to abstract your calls so the developer doesn’t need to know the SQL that is being run underneath. Allow simple complexity by allowing the developer to do so, but don’t require them to.

Allows non-routine problems to be solved easily. Always allow the developer to decide if their way is better. Provide a simple interface such that a developer can still use your engine to do things that you haven’t accounted for. If you can, try to make it easier. If you can’t, don’t stand in their way.

For a portion of my capstone, I have implemented what I believe is an excellent start of a DyDL, codenamed MagicDB.  I will be writing about this project more in the coming weeks, and will be showing bits of code, and explaining decisions made. Hope I don’t bore anyone.

coding, data, technology

Another call from the parents, wanting to get something working correctly on their computer.

You’d think we’d be past this… but we aren’t. That’s the problem with technology today. We are SO close to having things “just work”. In fact, I’ll admit, Apple is closer. The problem is, we aren’t there yet, even for the mac.

My parents are smart people. They don’t understand computers, but they are smart people. In fact, my father is one of the most intelligent people I know. He has a mechanical mind like none other. Once we start talking how to use the computer though, things get increasingly difficult. I wish I didn’t have such a short fuse, but I get frustrated easily with technical questions. Sorry guys. The problem still remains, when things get plugged in, and installed… they don’t always just work.

The reality is, will we ever get there? Sure, things are more and more user friendly, but sometimes the learning curve is just going to be difficult. Trying to explain how to get video conferencing is hard when the person who is being told is miles away and doesn’t understand why the computer needs to reboot to recognize the drivers, or where the unmute settings are.

This isn’t meant to be a rant - wait - I guess it is, but as software developers we always need to strive to make things “just work”. A lot of times, this isn’t the focus. Sometimes hardware is the limitation. Sometimes software is.  Sometimes the focus is quick development, and not quick learning curve. Sometimes we just specify a “requirement” on the system to avoid something that will cause the program to break. Although, every day we get closer!

What I worry about is that more and more, I need to explain to many of my students that things are “magic” and it just works. This is great - but it’s also scary. We need to be able to develop things that “just work”, but at the same time have a very clear and logical paradigm that is associated with them, so they can be understood. I don’t know how to solve this problem. Hell, I can barely think of examples. But, it’s been a month since I’ve written and I figured I’d say something.

Perhaps a few more days in the oven and this thought will be a little more cooked.

coding, technology

Recently I wrote a few new algorithms for a college blogging site that I am apart of called The College Blog Network. I went to lunch and had a few beers with Dan (the lead developer), and basically discussed with him the good and the bad of TCBN. The site is a great idea, and has a lot of potential, but we both agreed the scoring algorithms needed some work.

So, with a little discussion, I had come up with a couple algorithms. One of which has been implemented already, and one that will be implemented in the near future. I have outlined the general workings of these algorithms, but left out some of the details to prevent unforeseen exploitation.

Hot Topics Algorithm

The main goal of this algorithm is to score a blog entry based on how “hot” it is. Hotness is essentially popularity and freshness. Even if a posting was one of the most popular, 3 months later, it is no longer considered hot. At the same time, a post that has been popular for the last week should still be considered hot, even if it has been around for a few days.

The solution was simple, apply a weighting system to the score that is used to calculate “hotness”. The main factor is time or “freshness value”, which is an adjustable value that can be changed to accommodate TCBN as it grows.

As shown in the graph, full weighting will be applied to newer posts, but over time, the weight a post has will begin to degrade. Lets say your initial weighted score on your post is 10 initially. After X time, it will degrade to 5. To maintain a weighted score of 10, you will need a unweighted score of 20. This will allow for consistently popular posts to remain hot, but only initially popular posts to fade away quickly. Thus maintaining an accurate “hot list”.

Overall Blog Scoring Algorithm

The second algorithm I worked on is much more complex. The goal of this algorithm is to try to calculate the “goodness” of a blog. Currently the overall blog score is calculated by adding all blog “bumps” together, then removing the blog “dumps”. In the case of a tie, the blog “views” are used. While providing an decent ballpark estimate, this score is not easily changeable (you must specifically go out of your way to bump a blog), does not account for postings, and can be exploited positively and negatively if a group of people bump or dump all at once.

The main goal of this algorithm is provide accurate ratings for blogs. The requirements were that a visible score would still need to be generated, it would still be primarily based on user feedback (bumps/dumps), and would be less susceptible to exploitation.

Simply put, the new algorithm breaks down to this:

Base “Goodness” + Posting “Goodness” + Viewability

I can’t reveal all the details of how each value is calculated, but here is the gist of it:

Base Goodness is essentially the current rating system, with some minor changes to make it a little more robust.

Posting Goodness takes consideration of posts scores (which will be more apparent when post bumping and dumping is added in the next TCBN revision). This value considers all posts. If one post in a great while is “good”, your score will not be as high as if most posts are considered “good”.

Viewability is a non-user controlled value that takes unique post views into account. A post that never gets bumped, but is viewed a lot will contribute a small amount to the overall score of the blog.

Overall these algorithms were a fun little side project for me, and hopefully will come in great use for TCBN. Tweaking might occur once we start to see how the scores turn out, but these are the basic principles behind them.

coding, friends

Alright, here comes a rant. Sorry about this, but tutoring a student in PHP is painful. In fact, it’s not just PHP, but I am using PHP as an example.

Don’t get me wrong, I love PHP, but there is a single reason why tutoring people in it is so hard. The people.

People can need help with various technologies, and that’s fine. In fact, I really adore helping someone that knows how to program with a small problem. The problem with PHP is, most of the people that require assistance, don’t know how to program.

Setting people up for failure

PHP is used in the multimedia / web realm here at RIT. Sure, it’s used in database as well, but I rarely get asked for assistance from database students on how to use PHP, and when I do, it is more of a technical issue. The problem is, most students that focus in multimedia / web are not programmers. In the Information Technology program at RIT, you are required to take 3 courses in Java. Once you complete these courses, it is relatively easy to avoid programming in most of your courses for the rest of your tenure here. Students that did poorly in Java tend to go towards multimedia / web because it is easy. You can get away with almost no programming (writing HTML is not programming), and you are not forced to push yourself constantly when it comes to programming.

Enter PHP. As students hit the mid level courses, after trying to avoid programming for a long time, they have a new paradigm, stateless programming. Each time you post to a page, you lose all the data from the last page (unless you pass it along through POST or GET). Students struggle. Constantly, I have to work through basic programming concepts, and these frustrations are made worse by the lack of understanding of the state problem. Combine these weaknesses with a grasp of simple programming with the introduction of a database (which happens in some mid-level courses) only having phpMyAdmin as a guide, and students begin to drop like flies.

What is the problem?

Students pass the initial Java courses, and they are expected to be programmers. Unfortunately they aren’t. In fact, I know first hand that many students that finish the last course in Java couldn’t program anything remotely complicated if they were required to. Beyond this, with a weak foundation, they can not easily grasp another language. They never learned what makes a good programmer, and spend most of their time shotgunning their way through. Professors in later courses never cover the foundations again, and students get lost.

How do we solve it?

I suppose if I had a say, the Information Technology department would be much smaller. What it really comes down to is, students should be flunked in the early Java classes if they are not solid programmers. Yes, it’s a mean thing to say, but if you can not be solid at the core concepts, you are going to be struggling for the rest of your career. Programming is a major part of Information Technology. I will make a claim now that goes against what many of the “higher ups” in the department think, and what they would hate for me to say…

If you can not program, you will not be successful.

I admit, there are likely some people that couldn’t program their way out of a paper bag that are successful in the IT industry, but this is rare, and partly dependent on the definition of ’success’. Someone that gets a job isn’t necessarily successful (my definition may differ from others on this point). However, someone that can be innovative and inventive in the industry is what I would consider to be successful, and frankly, I can not see someone that can meet my criteria for success in IT without being at least a ‘decent’ programmer.

Whole rant aside, I want to make it clear that just because someone is in the IT program here at RIT, and focusing in multimedia / web, it doesn’t make them a slacker or bad at programming, because the concentration is, as with life, what you make of it.

coding, education, rambling, tutoring

For the last several months, I have been a programming and database tutor here at RIT in the Information Technology department. I have seen many students come and go for help, with varying levels of skill and motivation when it comes to programming.

I hate to say it, but I usually find my self disappointed in some students. I have been trying to figure out why they struggle with programming so much, and I want to try to help them get past the roadblock they are having… I have determined that one of the biggest roadblocks that students run into is having a profound lack of attention to detail and no patience.

Many times I get asked a question that could really be answered if someone just took some time to look at the code in detail, or think through a concept fully. I believe that people are fully capable of logical thought, so just taking some time is all that is really needed.

Step 1: Determine the problem you are trying to solve.

Even at the low academic level, you are programming for a reason. Before being able to code anything of value, you need to be able to abstract the fundamental essence of what it is you are trying to accomplish. If you are writing a program that will calculate area of shapes, that is one paradigm, another is connecting to a database. You must tune your brain to the frequency of the problem, and attack it from there.

Step 2: Develop a plan

Over the past few years, I have determined, I am not one to spend lots of time on design. In fact, I find that most design methodologies are too much overhead or too optimistic. I do however believe that you must always have a plan for what you are doing. A complex UML diagram is not needed, but a simple “WTF Napkin-gram” should usually suffice. Sure, part of this post is about detail, but the detail is most important in the code, not in the plan (this is my opinion… methodologists will wholeheartedly disagree).

Step 3: Code, and follow the plan

Here is where things start to go awry, and this is where I want to focus on a few key problems that I run in to.

People tend to write code blindly. You need to think about what you are writing. Every line should have purpose, and should be directly trying to accomplish the goals in your plan. Every line of code that you write that deviates from that plan is code that is considered ‘junk’. Look at the documentation for your language, and make sure you are not reinventing the wheel, but more importantly, understand what each function call you make does. Does it return a specialized object that behaves in a certain way? What are the bounds of your parameters? Take the time you need to make sure you understand each piece of what you write.

Focus on the error messages. A lot of the problems that I get asked to help with are (relatively) simple compile/syntax errors. Every language has some way that it can get these error messages to you, use them! This is where your attention to detail really comes in handy. You can look at a block of code and see things you might have missed, incomplete statements, and most common (but not always an error) incorrectly named variables. Make sure you understand the code you are touching, and be very explicit in fixing problems with it. Don’t shotgun debug, be precise about what you are trying to do,

Copying code is a bad idea… usually. This is one I am guilty of, in fact, most programmers are. There are several major flaws you need to worry about when you notice yourself copying code. The first problem is the most obvious, if you can copy code, there is a chance that the branch of code you are copying should be in a reusable function. Look at the code, see if you can abstract it to be a function call, and if so, do it! The second major problem with copying code is that its extremely easy to miss any changes you need to make to the code. For example, if you copy a few lines that are modifying variable xmlDoc, and you want the copied code to modify xslDoc, you might miss a change in going from xmlDoc to xslDoc. Happens all the time… so be careful. The third major point I want to touch on with copying code is this: I don’t recommend copying more then 3-5 lines of code at a time from another project. Be sure that you actually need the code you are copying. If you just grab an entire class file, you will likely need to change it enough that it might be a safer bet to just rewrite it… of course on that note…

Don’t reinvent the wheel. If code exists that does what you need, use it. In academia it is a little different, since we are trying to build up core concepts, but, in reality, if it has been done, you likely do not need to do it again. Many people could go on for hours about how important it is that programmers be lazy. I agree, programmers need to be lazy so they can see the way they can save time while coding, but remember that programmers also need to be proactive. You need to be able to spend time looking at documentation, and just playing with existing code to see what is already out there. Save yourself a headache later by taking some time to discover things now.

Program for programming’s sake. I guess this will be my last big point. A lot of people only code because they have to. Sure, this stuff isn’t for everyone, but if you want to be a good programmer, you need to do it a lot. Practice is how you begin to learn all the documentation, and all the tricks about a certain language (To Students: Yes, that is why I know where everything is in the javadocs). Really, the best thing I can recommend is to come up with a pet project, and do it. If it suits, trash the code and write it from scratch after you work out the initial kinks and know the problems you will run in to. The more you code, the better you will become.

These are only a FEW tips for becoming a good programmer. They are just some of the big issues that I see on a day to day basis with students here.

coding, education, rambling

This past quarter, I was taking a class in which I got to play with web services. I developed a web service in Java and used the latest application server, Glassfish, to deploy it. To consume the service, I was going to write a C# desktop application, and a Flash Lite mobile app. I thought it would be very exciting, and a lot of fun.

I was wrong.

The C# app was nothing remarkable. In fact, I was very impressed at how easy Microsoft has made web services. Just point Visual Studio at the auto generated (more on this later) WSDL file, and you can make the calls easily.

The flash application was…. more of a problem. So, in an attempt to explain why it was a problem, let me make a nice broad claim…

Flash will not work with Glassfish.

Random Yokel: But Dave! Isn’t the point of web services interoperability?
Dave: Why yes! Which is why I don’t know who to be more mad at…. Adobe or Sun!

So, Lets discuss what is wrong!

Using Glassfish, you automatically generate a WSDL based off of your Java code. The WSDL itself is valid XML, and a valid WSDL. However, when I pointed my little flash mobile test application at the WSDL, it failed. The error that Flash Lite 3.0 gave me was:

There are no valid services/ports in the WSDL file!

Weird. Especially because there definitely is a single, valid service and port in the WSDL… but that’s ok. I tried to turn it into a standard Flash application (rather than a Flash Lite 3.0). I removed the little bit of ‘mobile only’ code, and ran it again as a Flash 9 app (but the code was still ActionScript 2.0). It loads… so I try to execute my helloWorld() method (just returns the String “Hello World!”), and it fails:

Element tns:helloWorldResponse not resolvable

Strange. However, at least it appeared to get farther this time. Now, this error message gave me a little bit of a headache for a while… and then I found out the Flash does not support xsd:import. So, because the WSDL file that Glassfish will automatically generate (I suppose I should have guessed it would cause me problems), includes an import to an external schema defining your data types, Flash will choke. Ok. Let’s get around that. So, I took the WSDL that was generated by Glassfish and merged in the schema file. Simple enough, no problems. Run the code, pointing to the new WSDL which I hosted on my server (so: http://localhost/newWSDL.xml, rather then Glassfish app server: http://localhost:8080/Path/File?wsdl)… New error:

Unable to connect to endpoint: <PortURL>

Damn it. Nothing appears to be wrong with my code, C# and PHP can easily connect to my service. I was able to find this document, which basically says that Axis (another Sun app server) is sending poorly formed SOAP packets, and the solution is purchasing Flash Remoting. Right. I believe that. If the packets were poorly formed, I should be getting errors in other languages. Besides, my helloWorld() method is only returning a String! Personally, I want to think that it is some Flash sand-boxing issue, but I truly can not figure it out. Why this is so difficult? My final conclusion on this project was that Flash can not connect to a Glassfish app server (at least, out of the box). If someone has figured out how to make the two play nice, I would love to hear it.

This is bad news for Sun and Adobe. The point of web services is interoperability between systems. If you can’t deploy a simple application using Glassfish (which is the new Sun App Server…) that can be consumed by a simple Flash app… what is the point? Sure. I could go the Microsoft route, but what if I don’t want to? What if I can’t?

coding, education, technology

So, some people get all hyped up about Valentines day… other people don’t.

This comic is for the database geek that got all excited and swept up, and then realized the truth. Yes, I used most of the art for the last one, but give me a break! Finals are coming up!

Sometimes it just isn’t there. If it’s only there around February 14th… maybe it isn’t real :-P

coding, comic, relationships